R Programming: read.csv() skips lines unexpectedly

Question

I am trying to read a CSV file in R (under linux) using read.csv(). After the function gets completed I find that the number of lines read in R is less than the number of lines in CSV file (obtained by wc -l). Also, every time I read that specific CSV file always the same lines are getting skipped. I checked the formatting errors in CSV file but everything looks good.

But if I extract the lines being skipped into another CSV file, then R is able to read very lines from that file.

I am not able to find anywhere what my problem could be. Any help greatly appreciated.

Perhaps you could post some data to allow us to reproduce the problem? — Ari B. Friedman, Dec 19 '11 at 23:33
Here is the link to the CSV file I am trying to read - http://dl.dropbox.com/u/6967747/bugs.csv — Nitin Mohan, Dec 19 '11 at 23:35
Yup... quote="" works... Thanks for that. I did not try "" for quote but tried with other characters in them. — Nitin Mohan, Dec 20 '11 at 00:08
How come if I extract those skipped lines into a new CSV file, R reads the new one properly? — Nitin Mohan, Dec 20 '11 at 00:17
In particular, there are fields like `Mohammad Heydari (:rostam"` with unbalanced quotes which cause problems if quote delimited is expected. — Brian Diggs, Dec 20 '11 at 00:38

IRTFM · Accepted Answer · 2011-12-20T00:19:30.703

Here's an example of using count.fields to determine where to look and perhaps apply fixes. You have a modest number of lines that are 23 'fields' in width:

> table(count.fields("~/Downloads/bugs.csv", quote="", sep=","))
     2     23     30 
   502     10 136532 
> table(count.fields("~/Downloads/bugs.csv", sep=","))
# Just wanted to see if removing quote-recognition would help.... It didn't.
     2      4     10     12     20     22     23     25     28     30 
 11308     24     20     33    642    251     10      2    170 124584 
> which(count.fields("~/Downloads/bugs.csv", quote="", sep=",") == 23)
 [1] 104843 125158 127876 129734 130988 131456 132515 133048 136764
[10] 136765

I looked at the 23 with:

txt <-readLines("~/Downloads/bugs.csv")[
                 which(count.fields("~/Downloads/bugs.csv", quote="", sep=",") == 23)]

And they had octothorpes ("#", hash-signs) which are comment characters in R data parlance.

> table(count.fields("~/Downloads/bugs.csv", quote="", sep=",", comment.char=""))
    30 
137044

So.... use those settings in read.table and you should be "good to go".

Great demo of `count.fields`. – Richie Cotton Dec 20 '11 at 10:49 — Richie Cotton, Dec 20 '11 at 10:49

R Programming: read.csv() skips lines unexpectedly

1 Answers1

Linked