Submit a form as part of a cron job

Question

I have a url that I need to visit as part of a wider process on a project, I know that it works when I am logged in but obviously as part of the cron job it wouldn't be. If it were htaccess I would simply either use curl or wget and pass the username and password parameters accepted.

I have tried this already on this particular cron but it didn't seem to perform the task that the url is associated with. See example below:

curl -u username:password http://www.example.com (I would usually have the dev/null 2>&1 as part of the cron but I wish to see the output for now)

The problem is however that this page sits behind a form login and I am unsure of how to pass parameters to that form using a cron job.

Any help or advice would be greatly appreciated.

You can pass all parameters to form `action` url instead of going through form itself (form is only UI part for communication). — Justinas, Oct 08 '15 at 10:57
curl -d "username=[YOUR USERNAme]&password=[YOUR Password]" www.example.com/cronjob.php — Ninju, Oct 08 '15 at 10:58
You added the [php] tag, can you provide your PHP code? Did you try to submit and username and password when calling cron job script, and using `$_SERVER['PHP_AUTH_USER']` & `$_SERVER['PHP_AUTH_PW']` in its code ? — François, Oct 16 '15 at 02:25

score 0 · Accepted Answer · edited May 23 '17 at 12:09

Using Curl:

You will need to pass the form login parameters, probably using the POST method. Check the form's HTML to be sure.

To do a POST request with curl, see https://superuser.com/questions/149329/what-is-the-curl-command-line-syntax-to-do-a-post-request.

This might not work for some forms, which implement CSRF. To work-around this, you would need to parse the HTML, find the CSRF token, and pass it as one of the POST request's data parameters.

Next, the login most likely returns a cookie. Your browser normally saves this, and gives the cookie back to the website on each page request. You will need to specify a cookie file. See Send cookies with curl.

There may be some investigation to work-around any more complicated login schemes, depending on the website.

Using an automated web-browser

The much easier alternative, is to use an automated browser, like Selenium webdriver. There are scripting interfaces you can use, like Capybara (a ruby gem). Using Capybara and Selenium to control a browser, you can avoid any techniques that websites might have which makes using CURL difficult (eg. if they detect and block bots).

The disadvantage is that you need to install it. However, once you do, you can use simple commands to do stuff, eg visit('http://www.google.com'), click_link('Link Text'), ...

Also, see:

require 'capybara'

session = Capybara::Session.new(:webkit, my_rack_app)
session.within("//form[@id='session']") do
  session.fill_in 'Email', :with => 'user@example.com'
  session.fill_in 'Password', :with => 'password'
end
session.click_button 'Sign in'

Submit a form as part of a cron job

1 Answers1

Using Curl:

Using an automated web-browser