How to Download XML file from a web site with login

Question

I have a problem, I am very new to Java and I'm currently trying to create a Java application to download files from a website. To enter the website you need a password and username. I use NativeSwing to enter and get the path of the files, I saw many examples, but I don't know how implement them. Any advice ?

public class Test {
public JFrame frame;
private static JWebBrowser browser;

private static JPanel configurationButtonPanel; 

public Test() {
    frame = new JFrame("Test");
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    frame.add(createContent(), BorderLayout.CENTER);
    frame.setSize(800, 800);
    frame.setLocationByPlatform(true);
    frame.setVisible(true);
}

public JComponent createContent() {

    JPanel contentPane = new JPanel(new BorderLayout());
    JPanel configurationPanel = new JPanel(new BorderLayout());
    configurationButtonPanel = new JPanel(new FlowLayout(
            FlowLayout.CENTER, 0, 0));
    JButton beginButton = new JButton("Download");
    beginButton.addActionListener(new ActionListener() {

    public void actionPerformed(ActionEvent e) {
            ArrayList<String> ligas = new ArrayList<String>();
            int a=0;
            Document doc = Jsoup.parse(browser.getHTMLContent());
            Element ele = doc.getElementById("ctl00_MainContent_PnlResultados");
            System.setProperty("java.net.useSystemProxies", "true");
        try{
            Elements img_2 = ele.getElementsByClass("BtnDescarga");
            for (Element el : img_2) {
                for( Attribute attribute : el.attributes() )
                {
                    if( attribute.getKey().equalsIgnoreCase("onclick"))
                    {
                      ligas.add("https://portalcfdi.facturaelectronica.sat.gob.mx/"+attribute.getValue().substring(19,535));
                    }
                }
            }
        }
        catch(NullPointerException nulo){
        }
            for( int i = 0 ; i < ligas.size() ; i++ ) {
                  System.out.println( ligas.get( i ) );
            }
        }
    });

    browser = new JWebBrowser();
    browser.navigate("https://cfdiau.sat.gob.mx/nidp/app/login?id=SATUPCFDiCon&sid=0&option=credential&sid=0");
    configurationButtonPanel.add(beginButton);
    configurationButtonPanel.setVisible(true);
    configurationPanel.add(configurationButtonPanel, BorderLayout.NORTH);
    contentPane.add(configurationPanel, BorderLayout.SOUTH);
    contentPane.add(browser, BorderLayout.CENTER);
    return contentPane;
}
/**
 * @param args
 */
public static void main(String[] args) {
    NativeInterface.open();
    SwingUtilities.invokeLater(new Runnable() {
        public void run() {
            new Test();
        }
    });
    NativeInterface.runEventPump();
 }
}

When it run this part

 for( int i = 0 ; i < ligas.size() ; i++ ) {
     System.out.println( ligas.get( i ) );
 }

I get the following URL like this:

https://portalcfdi.facturaelectronica.sat.gob.mx/RecuperaCfdi.aspx?Datos=huswUYX1eXMlGkDiItMUBgaWREHHqhXOWtYxqyUh0oUZnCKLYE/gx6ENJ+0TwW5auWw8d/AiCJyuFSDNVY+5l0vkiELroo/fEmF+x5w+DQDDTfMX9qIINS1NgP9C1bFhirjcVXpZI1ed4ycpLPczkYMEGEKvqWemni8LWcbqC0BuZskOJnCQCaWRh1Kt7AL5GdBVKqkm3T5mvzhtkmE5dn0vcWbCFFO3d3G8hu7rlcc0XM+7+6iR52SZYYaHa/TOhcl2DjuzztADpa9tPxZ9VO6EzMVkYKTfDOqHwZO8m2U9BZ7UhFjqsyoAwsQneqhIqGwN21yEpGEcptsTb9uZ1t0Fc/1Ggd6SuK9NeGdBpiawn6cv6QM1uc4QQHMNpAgG89Rq5tOd4YAoRQHBe/vO8ppq60JwvJgQ4BN76EtZF0UtEWK+k57P01vatuvTHIdMBncbXyU+TrtE5AlhdGKkY2a8HwSxHw3nfoQ+SLBrjyg=

I think it's a really different question... Maybe the title it's wrong, because the real problem is to "Login to SAT and Download Files" ... The !"#!@% SAT it's "the Problem" (with P of $)#&"#$ ) — ThanatosMK, May 25 '15 at 22:41

Krzysztof Cichocki · Accepted Answer · 2015-05-22T08:02:49.840

They use applets for non-standard authentication, so take a look at my second post. This one is only valid for normal authentication mechanisms, i will leave it here, because it could be valid for someone else.

You need to connect to this site first, by sending the HTTPS authentication POST, with your credentials. You should do all this communication in the "https://...". The next step is to read a cookie value from the authentication response, and then set this cookie in to the header of the next URLconnection you will do for the resource. It is standard mechanism for server to distinguish the logged user, the authentication cookie is added to every request after login automatically by the browser, but in java, you need to do it manually.

It should work, if the server has a standard authentication mechanism, if it has different mechanism, you need to reverse enginerrig what is send, by using for example FireBug plugin, and switch to the Network tab, doing some log in, log out, downloading file, inspect what the communication looks like, and repeat it in your code.

Here you can find how to send your AUTHENTICATION POST with parameters in java: http://www.mkyong.com/java/how-to-send-http-request-getpost-in-java/

You will probably need to send parameters like password and login, but it could vary, look at the firebug network tab, when you login, it will give idea what params for that looks like. For example:

password=MySecretPassword&login=MyLogin

how to get cookies:

    URLConnection connection = new URL("your url").openConnection();

// insert auth parameters and set method to post
// call doInput, doOutput on connection
// read cookie

    List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
    myCookie = you need to change cookies list to one String containing all cookies, i don't have this code right now, but is should be easy.
    URLConnection resourceCon = new URL("your url").openConnection();
    resourceCon.setRequestProperty("Cookie", myCookie);

I'm stuck on this problem too... So far, I've been tracing every single step and request, but there is something I can't go forward... I have: 1) Request the login page, to obtain the session cookie, add it to my cookie handler so I can use it on further requests 2) Post my credentials, in exchange, I got a cookie with userId (added to handler) and a
on response that automatically submits again 3) Manually Post the form received using cookies, I receive a new
If I return the last form to my browser, It jumps to the page logged in. If I manually post the last form, I got error — ThanatosMK, May 25 '15 at 22:33

score 0 · Answer 2 · answered May 22 '15 at 07:59

I see, that they don't use standard authentication mechanism at this site:

https://cfdiau.sat.gob.mx/nidp/wsfed/ep?id=SATx509Custom&sid=0&option=credential&sid=0

They use some applets, you need to download the applets files that they have:

https://cfdiau.sat.gob.mx/nidp/applet/SgiCripto.jar and https://cfdiau.sat.gob.mx/nidp/applet/x509applet.jar

Then you could use some java decompiler, for instance this one: http://jd.benow.ca/

And after decompiling those project, you need to dig in to the code and do reverse-engineering to investigate how this site works.

How to Download XML file from a web site with login

2 Answers2