{{tag>Brouillon Web TLS}} # Note curl wget ## URL / URI Voir : * [Disséquons une URL, seconde partie](https://linuxfr.org/users/dascritch/journaux/cpu-ex0214-dissequons-une-url-seconde-partie) URI (Uniform Resource Identifier) : * Name : URN (Uniform Resource Name) * Location : URL (Uniform Resource Locator) ### URI URL, URI and URN ### URL En anglais, on la décrit ainsi : ~~~ scheme : // authority / path ? query # hash ~~~ Dans sa traduction française : ~~~ plan : // autorité / chemin ? requête # fragment ~~~ Enfin , comme on l'explique à notre manière : ~~~ comment : // chez_qui / où ? question # extrait ~~~ Source : https://cpu.dascritch.net/post/2023/10/12/Standard-%3A-%28Rappel%29-les-segments-d-une-URL Syntax of URL ~~~ scheme: subdomain/domain-name.Top-level-domain/sub-folder ~~~ ## wget / curl curl as wget ~~~bash curl -O -L http://acme.org/download/plop.tar.gz ~~~ Point wget to TOR/Privoxy: ~~~bash #export http_proxy=http://127.0.0.1:8118/ #export https_proxy=http://127.0.0.1:8118/ export ALL_PROXY=http://127.0.0.1:8118/ # Ou curl -x "http://127.0.0.1:8118" "http://gnu.org" ~~~ ## curl Voir aussi **HTTPie** Voir : * [Utiliser les REST API JSON avec cURL](http://www.zem.fr/rest-api-json-curl/) JSON ~~~bash curl -k https://aap-controller.local/api/v2/users/27/roles/ -X POST -u user:password --data-raw '{"id":534}' ~~~ ~~~bash curl --globoff --request POST \ --header "PRIVATE-TOKEN: " \ --url "https://gitlab.example.com/api/v4/projects/169/pipeline?ref=master&variables[0][key]=VAR1&variables[0][value]=hello&variables[1][key]=VAR2&variables[1][value]=world" curl --request POST \ --header "PRIVATE-TOKEN: " \ --header "Content-Type: application/json" \ --data '{ "ref": "master", "variables": [ {"key": "VAR1", "value": "hello"}, {"key": "VAR2", "value": "world"} ] }' \ --url "https://gitlab.example.com/api/v4/projects/169/pipeline" ~~~ Source : https://docs.gitlab.com/api/rest/ Préciser le host sans modifier les /etc/hosts ~~~bash curl -H "Host:www.acme.fr" 192.168.2.14:80 ~~~ Météo (Basé sur Wego) ~~~bash curl -4 wttr.in curl -H "Accept-Language: it" wttr.in/Rome ~~~ /etc/hosts ~~~bash curl --resolve www.acme.fr:80:192.168.2.10 \ --resolve www.acme.fr:443:192.168.2.10 \ http://www.acme.fr ~~~ Forcer SSLv3 Pour vérifier que ça ne marche pas, car ca chiffrement est obsolète `-k` Pour les certificat auto-signé Voir `CURLOPT_SSLVERSION` ~~~bash curl -3 -k --show-error --verbose https://acme.fr ~~~ Erreur status code ~~~bash curl --show-error --verbose -I http://www.acme.fr curl --write-out "\nhttp_code=%{http_code}\ total_time=%{time_total}\n" curl -s -o /dev/null -k -I -w "%{http_code}\n" https://www.acme.fr ~~~ Timeout ~~~bash curl -m 3 https://www.acme.fr ~~~ ### Fichier config .curlrc `.curlrc` ~~~ # Source : https://gist.github.com/v1m/f1d4751883f19c916515 # store the trace in curl_trace.txt file. beware that multiple executions of the curl command will overwrite this file --trace curl_trace.txt # store the header info in curl_headers.txt file. beware that multiple executions of the curl command will overwrite this file --dump-header curl_headers.txt #change the below referrer URL or comment it out entirely -e "https://www.google.com" #change the below useragent string. get your/other UA strings from http://www.useragentstring.com/ -A "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13" #some headers -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "Accept-Encoding: gzip, deflate, sdch" -H "Accept-Language: en-US,en;q=0.8" ~~~ `.curlrc` ~~~ Source https://github.com/necolas/dotfiles/blob/0d402c916f6e1118b20bd4befff863c1e4a2ee2d/shell/curlrc # Limit the time (in seconds) the connection to the server is allowed to take connect-timeout = 30 # Limit the time (in seconds) the whole operation is allowed to take # (prevents cURL from hanging due to slow networks or links going down) max-time = 300 # When following a redirect, automatically set the previous URL as referer referer = ";auto" # cURL default configuration file # http://curl.haxx.se/docs/manpage.html # Show error messages show-error # Show progress using a progress meter progress-bar # Send a fake UA string for the HTTP servers that sniff it user-agent = "Mozilla/5.0 Gecko" ~~~ Upload ~~~bash curl -v -u admin:admin123 --upload-file pom.xml http://localhost:8081/repository/maven-releases/org/foo/1.0/foo-1.0.pom ~~~ HTTP POST files here ~~~bash curl --proxy 127.0.0.1:4444 -F'file=@yourfile.png' http://0xff.i2p ~~~ Or with status bar ~~~bash curl --proxy 127.0.0.1:4444 -T yourfile.png http://0xff.i2p | tee ~~~ Upload command output ~~~bash command | curl -T - --proxy 127.0.0.1:4444 http://0xff.i2p ~~~ Upload command output with stderr ~~~bash command 2>&1 | curl -T - --proxy 127.0.0.1:4444 http://0xff.i2p ~~~ Or you can shorten URLs ~~~bash curl --proxy 127.0.0.1:4444 -F'shorten=http://example.com/some/long/url' http://0xff.i2p ~~~ ## wget Autres ~~~bash wget --no-cache --no-cookie ~~~ #### wget - preserve filename ~~~bash wget https://getsession.org/linux # Saving to: 'linux' wget --content-disposition https://getsession.org/linux # Saving to: 'session-desktop-linux-x86_64-1.8.6.AppImage' wget -O plop https://getsession.org/linux # Saving to: ‘plop' ~~~ #### Autres Fix curl client certificate error: curl: (58) unable to set private key file http://honglus.blogspot.fr/2012/03/fix-curl-client-certificate-error-curl.html 15 Practical Linux cURL Command Examples (cURL Download Examples) http://www.thegeekstuff.com/2012/04/curl-examples/ Download the webpage with wget. The wget options (-E, -k, etc.) are CASE SENSITIVE! The maximum depth to which the retrieval may descend is specified with the ‘-l’ option: ~~~bash wget -E -k -p -nd -r -l 1 -np -P http://site-to-be-mirrored.org ~~~ wget : Ignore robots restrictions ~~~bash wget -e robots=off --wait 1 http://your.site.here ~~~ Dossier répertoire directory ~~~bash wget http://www.claudiokuenzler.com/nagios-plugins/check_equallogic.sh -P /usr/lib/nagios/plugins/ ~~~ A partir d'un fichier d'URL ~~~bash wget --no-proxy -i providers.txt -P /usr/share/terraform/providers/registry.terraform.io/hashicorp/vsphere ~~~ ### Notes exemple Dokuwiki wget ~~~bash wget --max-redirect=4 --output-document=siteexport.zip --post-data="id=dokuwiki&depth=0&depthType=0.0&do=siteexport&ens=dokuwiki&template=dokuwiki" https://acme.com/doku.php?id=doku.php --http-user=USER --http-passwd=PASSWD ~~~ ~~~bash wget -E -r http://wiki.foo.tld/doku.php?id=start -l 0 --header="X-DokuWiki-Do: export_xhtml" ~~~ curl ~~~bash curl -L --max-redirs 46 -o siteexport.zip -d "id=dokuwiki&depth=0&depthType=1.0&do=siteexport&ens=dokuwiki&exportbody=1&renderer=&template=dokuwiki" https://acme.com/doku.php?id=doku.php --anyauth --user USER:PASSWD ~~~ ### Notes Curl certificat auto-signé Pour ignoré le certificat ~~~bash curl -k https://www.acme.fr wget --no-check-certificate https://www.acme.fr ~~~ Pour enregistrer le certificat et vérifier ~~~bash openssl s_client -showcerts -connect www.acme.fr:443 -servername www.acme.fr cacert.pem # ou curl -k -w %{certs} https://www.acme.fr > cacert.pem ~~~ Tester ~~~bash curl --cacert cacert.pem https://www.acme.fr ~~~ ou avec wget ~~~bash wget --ca-certificate=cert.pem https://www.acme.fr ~~~ Voir * [[Notes SSL_TLS HTTPS client OpenSSL]] * [[PKI - CA - Cert - Ajouter une autorité de certification]] Enlever la CA ~~~bash rm /usr/local/share/ca-certificates/acme.fr.crt update-ca-certificates --fresh ~~~ TODO : equivalement pour Redhat avec `update-ca-trust` `.curlrc` ~~~bash cacert=/home/jean/cert.pem ~~~ Ou, à éviter mais possible `.curlrc` ~~~bash insecure ~~~ https sous java Voir la paquet **ca-certificates-java** ~~~bash openssl s_client -showcerts -connect 192.168.4.13:5000 cert.crt # test curl --show-error --verbose --cacert cert.crt https://172.18.51.220:5000 cp cert.crt /usr/local/share/ca-certificates/acme.fr.crt update-ca-certificates /var/lib/dpkg/info/ca-certificates-java.postinst configure systemctl restart jenkins ~~~ ## Python ~~~python import requests # No warn for bad https 'verify=False' requests.packages.urllib3.disable_warnings() # HTTP GET all users ids r = requests.get('https://aap.admin.acme.local/api/v2/users/', auth=(aap_user, aap_pass), verify=False) ~~~ ## Exemple HTTPS certificat client ~~~bash curl --verbose --show-error --tlsv1.2 --cert /etc/httpd/ssl/acme.fr.crt --key /etc/httpd/ssl/acme.fr.key https://acme.net/plop wget --debug --verbose --no-check-certificate --secure-protocol=TLSv1_2 --certificate=/etc/httpd/ssl/acme.fr.crt --private-key=/etc/httpd/ssl/acme.fr.key https://acme.net/plop (echo -ne "GET /plop HTTP/1.1\r\nHost: acme.net\r\n\r\n" ; cat ) | openssl s_client -connect acme.net:443 -servername acme.net -cert /etc/httpd/ssl/acme.fr.crt -key /etc/httpd/ssl/acme.fr.key ~~~ ~~~bash curl --cacert keys/ca.crt --key keys/client1.key --cert keys/client1.crt https://pkiweb.lan wget --ca-certificate=keys/ca.crt --private-key=keys/client1.key --certificate=keys/client1.crt https://pkiweb.lan ~~~ ## Envoyer un fichier en ligne de commande (upload web) [Goploader](https://up.depado.eu/#cur) ~~~bash # Your file will be named myfile.txt curl -F file=@myfile.txt https://up.depado.eu/ # Your file will be named myamazingfile! curl -F name="myamazingfile!" -F file=@myfile.txt https://up.depado.eu/ # Reads data from stdin cat myfile.txt | curl -F name="stdin" -F file=@_ https://up.depado.eu/ ~~~ ## Sécurité Pipe bash https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ ~~~bash curl -sSk http://pluver.xqi.cc/setup.bash curl -sSk http://pluver.xqi.cc/setup.bash | bash ~~~