7

I need to download contact data via a REST API, which I get in JSON format. Issue is, it might be many many contacts, so I want to observe the progress (how many contacts have already been downloaded) and report back to the user (with a progress bar, the code below runs in a thread).

However, it seems that the line client.execute(getRequest); establishes the connection and downloads the whole content in one go, i.e. my InputStream reader (to fetch the content in chunks) seems to be useless. Now I'm wondering how to make it work in chunks, so I can report on the progress?

/** prepare HTML get request */
HttpGet getRequest = new HttpGet(url[0]);
getRequest.addHeader("Authorization", "OAuth " + myTokens.get_access_token());

/** execute HTML request */
DefaultHttpClient client = new DefaultHttpClient();
JSONArray records = null;
HttpResponse response = client.execute(getRequest);

/** init response handlers for input stream */
InputStream input = new BufferedInputStream(response.getEntity().getContent());

byte data[] = new byte[MAX_BUFFER_SIZE];

long totalContactsCount = -1;
int readContactsCount = 0;
int currentByteReadCount = 0;

/** read response from inpus stream */
while ((currentByteReadCount = input.read(data)) != -1) {
    String readData = new String(data, 0, currentByteReadCount);
    dataString.append(readData);

    // then +1 progress on every ...},{... (JSON object separator)
    if (readData.indexOf("},{") >= 0) {
        readContactsCount++;
    }

    // publishing the progress....
    if (totalContactsCount > 0) {
        publishProgress((int)(readContactsCount * 100 / totalContactsCount));
    }
}
input.close();

/** transform response into JSONArray */
String result = dataString.toString();

//... convert into JSONArray
benfwirtz
  • 271
  • 6
  • 16
  • What if there is more than one '},{' in the buffer you read? – Diego Sevilla Jul 14 '11 at 23:33
  • You are right - I simplified the code here, but what will happen is that if the latest read includes multiple contacts, the progressbar in the UI will not get to 100% (the code will continue to read the inputstream though). However, the issue remains that the download seems to happen before I open the InputStream. – benfwirtz Jul 14 '11 at 23:36

2 Answers2

2

Unfortunatelly, as it seems from the documentation on DefaultHttpClient, the execute method returns a HTTP response, already parsed, and hence, already read from the network, so you cannot plug in the input stream. That stream is prepared for you to be read, but once it has been retreived, as you have noticed. You'd have to use another strategy. For example, trying yourself the connection using the sockets API.

Diego Sevilla
  • 28,636
  • 4
  • 59
  • 87
  • Thanks. I tried the socket approach, but it also works (a bit simpler) with HttpUrlConnections. `URL urlObject = new URL(url[0]);` `HttpURLConnection conn = (HttpURLConnection) urlObject.openConnection();` `conn.addRequestProperty("Authorization", "OAuth " + myTokens.get_access_token());` `InputStream input = conn.getInputStream();` – benfwirtz Jul 16 '11 at 15:48
  • Ah, perfect. Hooray for `HttpURLConnection`! :) – Diego Sevilla Jul 16 '11 at 16:40
1

You can first ask the web service for the total number of contacts it has (say 100,000), and then ask it for individual "pages" of contacts (say 100 contacts/page). As each page arrives, you update the progress bar. If a page request fails just retry that request and continue processing. This should be a much more reliable and efficient approach than trying to get all 100k contacts in one massive request.

Jim Ferrans
  • 30,582
  • 12
  • 56
  • 83
  • Great idea ! I was able to solve it another way, but this is definitely a concept I might use in another app - thanks ! – benfwirtz Jul 16 '11 at 16:02