The Hardest Part About Analyzing Your Data, Is Now Gone...

6/7/2016
 

Let's start with a look at Visual Analysis.

Cooper, Heron & Heward (2007, p. 149) say if you do visual analysis, you must look at three things.

  • Trends of data
  • The level of data
  • The extent in type of variability and data

And here is a very nice quote from Lane & Gast (2014, p. 445):

"You need to look at variability or stability, you need to look at level, and you need to look at trend."
 
Do I have to convince anybody that if you are doing visual analysis, these things are important?
 
You probably agree with me. This is just how we do our business.

Trend
Take a look at the first graph below. We have an ascending trend, then we have another ascending trend. How many would be convinced that the intervention was good? That the progress happened as a result of the intervention itself? Would you be suspicious?

Picture
 

You should be! The problem is the behavior is already going up. So if something's going up and you say, "I did this thing and now it's going up," who are you convincing? This is done completely with visual analysis. You don't need statistics for this. You can look at the data and see. This is our science.

Now look at the second graph. If it's flat and then it goes up, that's convincing. And the third graph? If we have something that's going down and our intervention makes it go up, we can convince people of our effectiveness. This is part of trend analysis.

Level
Let's just say we have the average in baseline, which is a 2. Then you do something, now the average is a 10. Do we have an effect? Yes! You went from 2 to 10.
 
You're doing this through visual analysis.

Variability
Imagine that you have data points that are bouncing around and those two lines below are showing you what's called the variability envelope.

Picture
 

On the left, although it went up, the variability envelope is the same. Has variability changed? No. On the right, it has.
 
So here's my question to you:

Where does visual analysis occur?

Laboratory settings. Maybe you do work in labs and do visual analysis there. Maybe you are doing applied work where, again, visual analysis is frequent. You did a thesis, you did a dissertation? You probably did visual analysis.

Now, here is a critical question:

What does visual analysis rely upon?

Graphs.

Think about this for one minute. All that stuff about trend, level, variability; if you have a compromised visual graphic, how good is your analysis? It will likely also be negatively affected. I can't tell you how important it is that we have a very good graph when we're making decisions. All of those things that I just shared with you; how we move our science forward, how we make applied decisions, how we affect lives. Everything is reliant upon a graph.
 
So would you believe no one has ever done a study, not once, ever, to see how well we actually make graphs? No one's ever done it. I don't know why. But I decided that's something I needed to do. More on that later...

If you're ahead of the curve and want to get started, Chartlytics enables you to create the most effective graphs for monitoring and changing behavior. Find out how it works here, or sign up for a free trial.  If you’re ready to really dig in and move at your own pace, head over to Precision Teaching University.

Also, tell us about your graphing pet peeves on Facebook and Twitter.

You're Doing Your Client a Disservice Unless You Learn These Principles of Graphing.

6/6/2016
 

Get the video here at Precision Teaching University 

Speech Transcript:

Emcee: Hello, everybody, welcome back. I'd like to introduce our next speaker, Rick [inaudible 00:00:16].

Rick: Yeah. Rick, please.

Emcee: Let's kind of [inaudible 00:00:23] settle down. Rick is a professor of special education at [inaudible 00:00:29] New York City. He teaches courses on methods for teaching reading, informal assessment, [inaudible 00:00:36] single case design. Rick conducts [inaudible from 00:00:40 to 00:00:49] special education journalists. He's was the past editor of The Journal of Precision Teaching Celeration. Let's welcome Rick.

Rick: Thank you. As you can tell by my playful slide, what I'm about to share with you today is received in one of two ways. One way, it becomes uncomfortable for some people, and the other way, it's very welcoming to some people. This research that I did gets at the heart of something all of us do every single day when it comes to understanding if we have an effect. And that's called visual analysis.

I submitted to major behavior analytic journals, and you would think that I was saying, "Let's go mentalism." They attacked my paper. Two reviewers just refused, they said, "You can't ask that question." They literally said, we couldn't ask my question. We rejected it based on them not being able to understand what we were asking, and I was incensed, so I mailed the editor, and the editor agreed with them.

And you would know these journals, you would know these people. But I get it. I get why this can strike a nerve. The good news is, my paper is published. But the only way I can get it published is if I went outside of the behavior analytic community. So I went to an ed psych journal and they published first go-around.

What is this controversial thing that I'm going to talk to you about today? It's graphing. And you can tell by the title here that things aren't so good. But before I get into that, let's talk about visual analysis. Visual analysis, this comes from. They say that visual analysis is the extent in type of variability and data, number one. The level of data, number two. And then trends of data. If you do the visual analysis, you must do these three things.

Here is another very nice quote, and this comes from Lane and Gast, and now you're talking about visual analysis. And they say that it's the cornerstone, the cornerstone of single case experimental design. And then they say, again, "You need to look at variability or stability, you need to look at level, and you need to look at trend." Do I have to convince anybody in here that if you are doing visual analysis, these things are important?

You probably all agree with me. I don't have this...all of our textbooks say that. This is just how we do our business. We take data...again, I'm talking about time series data here, and I'll talk about that soon. But if you're having time series data, you're looking at the variability, the level, and stability.
So take a look at this first graph. Here, we have an ascending trend. Then we have an ascending trend. How many would be convinced that the intervention was good? That the intervention happened as a result of its own? Would you be suspicious? Why?
Man: The market's going up.

Rick: Exactly. It's already going up. So if something's going up and then you say, "I did this thing and it's going up," who are you convincing? This is done completely with visual analysis. You don't need statistics for this. You can look at the data and see. This is our science. So here, if it's flat and then it goes up, that's convincing. If we have something that's going down and we want it to go up, we can convince people. This is part of trend analysis.

And then there's something called level. If you have ten data points, there's a way to figure out what's the average of those ten. And there's different ways...you could just take the arithmetical average where you add them all up and divide by ten. You could take the median, which is what a lot of people recommend. And then that would become your level or your average. You could take the geometric mean, which is what I would recommend that other people do.

But let's just say, here, we have the average and baseline, which is a 2. Then you do something, now the average is a 10. Do we have an effect? Heads up and down. Yes, we have an effect. You went from 2 to 10. On average, you have an impact. How about here? Yeah, at best, you could say, "We have the most modest, mild effect." That's not something you're going to write home to Mom about, right? Really non-impressive there.

You're doing this through visual analysis. And then the last part of this would be the stability or the variability of the data. And imagine that you have data points that are bouncing around, and those two lines right here are just showing you...that's called the variability envelope. Here, although it went up, the variability envelope is the same. Has variability changed? No.

But look here. Has variability changed? Again, visual analysis...this is what we all need to be doing. This is what our textbooks say. This is when you read anything in our prominent journals, this is what people are doing. Or at least they should be doing.

So here's my question to you. Where does visual analysis occur? Well, this is the first question. Laboratory settings. Many of you have worked in labs or maybe you do work in labs, we do visual analysis there. A lot of you are doing applied work. Visual analysis, frequent. You did a thesis, you did a dissertation? You probably did visual analysis.

How about this conference? You go and see speakers who are talking about time series, you go to ABAI and you look at the poster sessions...you're swimming in this data. Our journals, this is how we define treatment effects, intervention effects, visual analysis. In other words, everywhere, you're going to have these time series data.

Now, here is the thing that is a critical question. What does visual analysis rely upon?

Woman: Graphs?

Rick: Yes, thank you whoever said that over there. Visual displays. Graphs. Think about this for one minute. All that stuff about trend level variability; if you have a compromised visual graphic, how good is your analysis? It will likely also be negatively affected. I can't tell you how important it is that we have a very good graph that we're making decisions upon. All of those things that I just shared for you; how we move our science forward, how we make applied decisions, how we affect lives. Everything is reliant upon that graph.

So what did I do? I wanted to know how well do we do that as a field? Would you believe no one has ever done a study, not once, ever, to see how well do we actually do our graphs? No one's ever done it. I don't know why. But I decided that's something I need to do.

Now, if you're going to do a study about graphs, you have to all understand that there are rules for constructing your graphs. I didn't make up these rules. So don't get angry at me when I tell you what these rules are. A lot of people don't know about all these rules. You can go back to 1915 in the American Statistical Association...and by the way, when you go back in time, the American Statistical Association had some really interesting people. And their take back then, in the early 1900s on visual graphics, very different from what you would get from a modern-day statistician. Now, graphs are looked at as superfluous and, "Why do you need it if you have numbers?" But back then, it was a very different time.

Back in '38, there was the American Standards Association. This was a bunch of business leaders, a bunch of mechanical engineers, people from statistics. In '60 and '79, the National Standards Institute, American Society for Mechanical Engineers. '88, there's the Scientific Illustration Committee, and even the Department of the Army came up with rules. And of course, when you look at the ones that are through time, you can see that they reference other sources. So there are rules.

Where did these rules come from? These rules came from these people that designed the graphs. When you have a visual display, it's meant to do something. It's meant to tell you information. And depending on the type of visual display you're using, then there are rules for that construction.
How about in our field? Did any of you ever read this article right here by Parsons? This should be standard reading for everyone. It's in a chapter, and it's just a wonderful exposition of the rules and all of the things that we should be looking at for graphical display. Polling has nice information in his texts. Again, these are rules for graph construction. And of course, you've all probably read Cooper, Heron & Heward. Outside of our field...we didn't invent line graphs. We didn't do it. These three texts right here are people that are citing those sources that I shared with you, and there's more sources than that.

So I took all of these rules, number one, and I had to figure out, "What are all..." there's a lot of rules. In fact, if you went back in time, many people that would do graphs would send it off to a drafts person. A person highly skilled in the technical specifications of creating these graphs would create a graph. Now, with Excel, everybody does them. If you have access to a computer, you can whip up your own graphs.

How many of you were taught the rules of graphing? Where did you get those rules? From Cooper? Okay. So many of you, that was a lot of hands, which is excellent. You understand these rules. So I looked at these rules, and everything that is in Cooper, Heron & Heward is not everything that's in all the rules, right? A lot of them...the important ones are there. But there are more.

And so, our question was, "How well do these selected visual graphics that we find follow what's called the 'essential structure' and 'quality features'"? The essential structure means if you don't have these things, you don't have a time series graph, period. And then quality features would be other features that enhance the usefulness of you understanding your data.
I always get a kick out of people when they're negative about graphs, and they're like, "Oh, we had these numbers, and those are just very simple..." there's so much complexity and elegance when you have time series moving through time. People just don't understand that. But we do have that. These quality features are important.

We followed...there are procedures out there for selecting the journals that we pick, because we have a lot of behavioral journals, more are coming on the scene all the time. So Carr and Brittain [SP], Critchfield, and then myself and colleagues, we have these criteria. And that led us to selecting eleven behavioral journals.

Tell me if you've heard of these journals before. "Behavior Modification," "Behavior Therapy," "Child and Family Behavior Therapy." "Cognitive and Behavioral Practices"? Yes, it is a behavioral journal. "Journal of Applied Behavior Analysis." "Journal of Behavior Therapy and Experimental Psychiatry." "Education and Treatment of Children." "Journal of Behavioral Education." "Journal of the Experimental Analysis of Behavior." "Learning Behavior," this used to be called "Animal Learning and Behavior." "The Analysis of Verbal Behavior."

You've heard of those journals. If not all of them, probably most of them. Our field is not static. We have people doing applied work. We have people doing educational work. We have people doing clinical...we just have all...our rich science is moving into all of these different areas. And we need journals to be able to accommodate the things that we're interested in in talking about.

So what we did is we took all of these journals, and we went back to the day that they were started. Java, 1968. So that meant that every two years, we would take a random issue, and then that's how we came up with all of these graphs.

We used all line graphs. If you had dual vertical axes, we excluded it. If you had a nominal or ordinal unit, like, you named something on the vertical access, that was excluded. If you had a non-time unit, that was excluded. And of course, if it was logarithmically scaled, we excluded that. We ended up with 4,300 journals. That's a lot of graphs to go over and apply these to. And it was very difficult, but by gosh, that team of graduate students persevered, and they did it after about a year's worth of time. And now, I'm here to tell you about that.

So what are the results? Let's go over what some of these things that we looked at and what's some of the things that we found. When you have graphs, a line graph...and again, I'm talking about time series graphs. That word, "time series," what does that mean? It means that you collect data, and that data marches through time. That time can be minutes, it can be hours, it can be days. It's some time unit that your behavior is changing. Now, sometimes, we have colleagues that would share a scatter plot. That's not a time series data. If you have a bar chart, that's not a time series data. There's a lot of things we do that aren't necessarily time series. So I'm everything I'm talking to you about today deals with time series.

But what's the most popular graph that we use as a field? It's this. They're everywhere. If you have a line graph, it has to have, at minimum, a vertical and horizontal axis. I'm going to refer back to this frequently. This is a prototypical graph. And what you can see here is you have a vertical axis, and you have a horizontal access. Take a look at this graph. Do you notice what's missing? They decided, "We don't need a vertical axis.

Here's the results. Ninety-eight percent of those 4,300 graphs we looked had a vertical access. That's pretty good. Ninety-eight percent. Ninety-seven percent had a horizontal axis. That means a label...sorry, a horizontal axis. That meant that 2 to 3% error rate. You may be thinking as I present to you, "What's an acceptable error rate?" I don't know. No one's ever done this study before. What do you think? I'd like to hold us up to a standard of, "We should never have an error."
There's one other study that I looked at...you may be familiar with one of the most prominent journals in the world, which is called "Science." And this person named Cleveland went through science and he looked at what kind of errors...he wasn't looking at line graphs, but he [inaudible 00:17:25], and people make errors in graphs. And his rate was anywhere from 5 to 10%, depending on the things that he looked at. I don't know what the acceptable rate would be.
Vertical and horizontal axes have to have labels. When you look at the graph, you need to know "What's going on here?" So the way it works for line graphs, you have a quantitative amount here on the vertical axis, and then you have a unit of time. You have to have a unit of time. Because what are these graphs? They're time series. So you have to have a unit of time.

Here's a graph, and they have labels, but what happens there? How do you know what that is? What do you have to do? The only way you can know is if you go in the journal and actually read it, and hope that that information is in there. Tuki [SP], who is a great data scientist, he did a lot of stats, but he also talked a lot about visual display. He said, "A good graph should show all the information just when you look at." And that's true. And one of the fundamental parts of that is having labels on our graphs.

Another issue, which you may be familiar with, is sessions. Sessions is a...it's not a time unit. We included this because so many people use sessions. But that also occurs often. So what are the results here? Eighty-three percent of vertical axes had a label. That meant that 17% of the articles re-reviewed didn't have a vertical axis label. That's a pretty big number when you think about it. Thirty-one percent had a time unit on the label. Thirty-one percent. That meant that almost 70% of the articles we looked at either had no label at all, or they had what's called a non-time unit label. Almost 70%.

Have any of you heard of the National Institute of Standards and Technology? You may have. They have what's called the Office of Weights and Measures. And what they do is they're designed to do one thing, and that is to promote uniformity among measures. And you've seen this...you live your lives with the decisions that they make. And they define time as seconds, minutes, hours, days, weeks, months, and years. Guess what's missing?

Man: Sessions.

Rick: Yes. Sessions is not a unit of time. Don't use it. It's really amazing to me that we have all of these graphs with sessions. Doesn't it matter to you if we do an intervention and it happens in one week, one month, or one year? Don't you deserve that information? You do. I do. Our consumers do. When you use sessions, you just say, "You know what? You don't deserve that information. Because you have no way of knowing." You could say, hour sessions, hour non-consecutive sessions. Tell us the time unit. Very important.

Here's another aspect of creating journals, it's called the Proportional Construction Rule. Some people also call it the Two-Thirds Rule or the Three-Fourths Rule. When you look at how graphs are created, there's a vertical axis and there's a horizontal axis. When you look at those two axes and you put those together, do you see the proportion of the vertical to the horizontal? It needs to be two-thirds or three-fourths...that's 66 to 75% the size of that.

Why is that rule there? This top graph is created with the Proportional Construction Rule, and then you can see the slope of the line. And there are very technical reasons for why it should be two-thirds, which I won't get into here. But look what happens when you decide to stretch the vertical axis. What does it do with the slope? You've just exaggerated your data. You just said, "Oh, look at this. It's really good."

What happens if you squish, you compress the vertical axis? What does it do with the slope now? It depresses it. Think about that for a moment. We have a field, we have an army of people out there doing this stuff, like, silly-puttying their graphs. How is that good for any of us to be doing that? Here's a graph that I'll show you. There's the vertical, there's the horizontal, and what do you notice? That wasn't three-fourths.

What are the results? Only 15% of the graphs that we reviewed follow that rule. That means that 85% of the graphs...that means, really, nine in ten people are not making [inaudible 00:22:23]. If that's true, nine out of ten of you are not following this rule, if that generally holds up. That is something that we must address. If we're going to continue to use linear graphs...that's the second part of my talk. But if we're going to continue using these linear graphs, we must form our graphs properly. Otherwise, we're telling lies. We're exaggerating things.

And I'm not saying we're doing this on purpose. A lot of people don't know better. In fact, I've actually found some single case design books in some journals where they encourage people to do that. They say, "Oh, do you want to see the effect? Well, you should really stretch that out to show the effect." That's exaggerating our data. We should never be doing that. We need to understand what the data is telling us and react to that appropriately.

How about tick marks? This may seem mundane, but graphs need to have tick marks. Tick marks help you understand the data because data are occurring in time, and there's also a value. Tick marks help you orient...you form this understanding of what the data are telling you. Some people don't believe in using tick marks, as in reference to this article.

So when we looked at our graphs, 78% had tick marks. That meant that 22% of graphs either didn't have vertical tick marks, horizontal axis tick marks, or no tick marks at all. That may seem like a minor thing, but that's an error. You should not be constructing graphs that don't have tick marks.
Let's talk more about the tick marks. Did you know the tick marks are supposed to be pointed out? Not over, not in, out. It seems like, "Wow, I never knew that." Here, you can see the tick marks are pointed out. If you look at these tick marks here, you'll notice...and the reason why that rule is, is it helps to make a busy chart. Anything that detracts from you being able to extract the story of the data is not good. Tick mark's pointing in, it may seem minor, but that's a rule. So we looked at how many tick marks were pointing out. We found only 43% of the journals that we examined followed that rule. Which meant 57% of the graphs had tick marks that were pointing inward or above those axes.

Condition labels. What's happening with the data when you do it? Here, we have a condition, it's called baseline. Here, we have another condition, it's the intervention. You all do things; maybe you have a baseline, you have different interventions you do. If you're going to share that information with anybody, you need to let us know what you're doing. Otherwise, we don't know.

Here is a graph and if you look at it, you don't see anything. But if you read the article, what you find out is there's not only condition lines, there are no condition labels, either. So I went in...you have to go in, and you can see what they're doing. That's not a good graph if you don't have your condition labels, and you're not letting the person know what are the differences. Because otherwise, then what you're doing is you have to go back and forth between the text and that visual picture. That visual picture should be striking. It should speak to you. It should be evident what happened. That's what our science is built upon. That's what we all need to aspire to.

The results? Ninety-three percent of graphs had labels, which means 7% of those graphs didn't have labels. Here's something that should be very obvious, and I don't always get angry when I get rejected, because if you're in higher-ed...if you publish things, you're going to get rejected almost all the time. Very few people are going to have most of their things. So I'm okay with it, because that's part of science. You take your data, you give it to your peers, they critically review it, and then you move forward.

But the journal editor and these reviewers really bothered me because I'm so passionate about our field. And the fact that these big-name people were saying things like...here's why two of these people rejected my paper. They said, "There's no evidence for what you're saying." They said because I didn't have evidence that you have to have this construction...because I didn't have studies that showed what visual effect that had, they wouldn't review my article. I'm like, "That's another study. I didn't ask that question." There are these rules over here, and my question was, "Do people follow these rules?"

But one of the things that I argued, I said, "Okay, yes, that's something that should happen. There should be a whole arm of behavior analysis that studies just this graphics." And I argued, I said, "Okay, so you mean to tell me that I actually need data to convince you that our data points needed to be legible on a graph?" That's how ridiculous these reviewers, in my estimation, were. They just couldn't handle the fact that all of these results I'm sharing with you, they don't point out...it doesn't paint a good picture of what our graphs look like in journals.

So there's our data points. These are very clear. How clear are those data points? They're not even there. You have data paths is all you have. I want to see the data points. You should want to see the data points. So what are the results for...these are legible? Well, 86% of the data points were legible, which meant 14% of the data points reviewed, that's a big number, 14% of 4,300.
I did other things that...I'm not going to walk you through all...because you can start looking at comparing data and there's all kinds of rules that we should be following, and we did that in our paper. If any of you are interested, I'd be happy to share that with you.

But this impacted me, and it compels me to share this with anybody that'll listen. So thank you all for listening to this and not throwing tomatoes at me and walking out. You are...I feel if you're a scientist, you have to embrace the data. And if you don't like the data, that's okay. Science is all about...it's a marketplace of ideas. I do research on certain things. But you know what? If data came around and said, "That thing you're doing is not good," I wouldn't be doing that thing anymore. I'd be doing a new thing. That's what we have to do as scientists. We must accept what the scientific evidence tells us.

So where am I in my career? After doing this...and I'm going to continue doing research on graphs, you're probably saying, "Well, what is the solution to this?" There are problems and...



If you want more of Rick, but can’t make it to any of our webinars or events, check out a full online school set up and taught by Rick. http://www.precisionteachinguniversity.com/

 

Subscribe to Email Updates