{{tag>Brouillon Python CA}}

= Notes JSON et YAML


| **jc**        | JSON Convert JSONifies the output of many CLI tools and file-types             |
| **jq**        | Command-line JSON processor                                                    |
| **json_pp**   | JSON Pretty Printer                                                            |
| **JMESPath**  | JMESPath is a query language for JSON (utilisé par ''json_query'' d'Ansible    |
| **jp**        | The ''jp'' command is a command line interface to JMESPath                     |


Voir :
* JSONPath et go_template
* JSON RFC 6902 patch


Voir Ansible :
* https://blog.stephane-robert.info/post/ansible-filtres-advanced-2/
* https://docs.ansible.com/ansible/latest/collections/community/general/docsite/filter_guide_selecting_json_data.html

Voir aussi :
* [[https://mgree.github.io/ffs/|Monter un fichier JSON comme en système de fichier avec ffs]]


<code bash>
sudo apt-get install jc
dig www.google.com | jc --dig -p
</code>

<code javascript>
[
  {
    "id": 17598,
    "opcode": "QUERY",
    "status": "NOERROR",
    "flags": [
      "qr",
      "rd",
      "ra"
    ],
    "query_num": 1,
    "answer_num": 1,
    "authority_num": 0,
    "additional_num": 1,
    "opt_pseudosection": {
      "edns": {
        "version": 0,
        "flags": [],
        "udp": 4096
      }
    },
    "question": {
      "name": "www.google.com.",
      "class": "IN",
      "type": "A"
    },
    "answer": [
      {
        "name": "www.google.com.",
        "class": "IN",
        "type": "A",
        "ttl": 38,
        "data": "142.250.178.132"
      }
    ],
    "query_time": 8,
    "server": "208.67.222.123#53(208.67.222.123) (UDP)",
    "when": "Sun Oct 16 21:54:04 CEST 2022",
    "rcvd": 59,
    "when_epoch": 1665950044,
    "when_epoch_utc": null
  }
]
</code>


<code bash>
cat fic.json |python3 -m json.tool |native2ascii -encoding UTF-8 -reverse
</code>

Extension Firefox http://jsonview.com/

<code bash>
sudo apt-get install yajl-tools
</code>

yajl-tools
* json_reformat 
* json_verify

<code bash>
mongoimport --host localhost --db database --collection collection <fichier.json --jsonArray
</code>

Convert JSON to YAML
<code bash>
yq -P sample.json
</code>

Ou encore
<code bash>
python -c 'import sys, yaml, json; yaml.safe_dump(json.load(sys.stdin), sys.stdout, default_flow_style=False)' < /opt/stack/keystone/etc/policy.v3cloudsample.json > /etc/keystone/policy.yaml
</code>

<code bash>
pip install jsbeautifier
js-beautify file.js
</code>

Dico vers JSON ?
<code bash>
cat plop.t |tr \' \" |sed -e 's/None/""/g' |jq .
</code>


VIM - Formater le JSON
<code ->
:%!python -m json.tool
</code>


== Requête (Query) 

Avec curl
<code bash>
curl -k https://aap-controller.local/api/v2/users/27/roles/ -X POST -u user:password --data-raw '{"id":534}'
</code>

Avec **jq**
voir : https://hyperpolyglot.org/json

Top level sections
<code bash>
cat single-instance.json |jq 'keys[]'
</code>

Exemple wildcard
<code -> 
$ curl -s -u "${AAP_USER}:${AAP_PASS}" "https://aap.acme.local/api/v2/job_templates/" | jq '.results[0].id'
1680

$ curl -s -u "${AAP_USER}:${AAP_PASS}" "https://aap.acme.local/api/v2/job_templates/" | jq '.results[].id' | tail -3
1572
1569
2343
</code>

jq retours sans guillemet (quote) :
<code bash>
jq -r
</code>

<code bash>
docker inspect portainer/portainer-ce |jq '.[0].RepoTags'
</code>

<code bash>
ansible -i testsrv, -m setup all |sed -e 's/^.*=>//' |jq -r '.ansible_facts.ansible_mounts[] |.mount'
</code>


=== Filtres

Conversion string integer avec ''tonumber''

Exemple 
<code bash>
jq '.blockdevices[].size | tonumber'
</code>



== Autres

=== Ansible - Utilisation de boolean en extra vars via JSON

<code bash>
ansible-playbook playbook.yml -i inventory.ini -e '{ "remove_vhost": false }'
</code>


== YAML

| **yq**                                    | portable command-line YAML processor |
| **[[https://gitlab.com/t0pd4wn/yb|yb]]**  | YAML Parser in bash                  | 

Voir :
* https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.colons
* https://stackoverflow.com/questions/3790454/how-do-i-break-a-string-in-yaml-over-multiple-lines/21699210#21699210

Contrôle syntaxe : 
* [[https://morioh.com/p/4d20ac5ddcc0|Yamale]]

Ce que **jq** est à json, **yq** l'est pour le yaml.

**yq** permet également de convertir des fichiers JSON en YAML
<code bash>
cat .kube/config |egrep -v "\-data"  |yq .
</code>

<code bash>
# pip install --user yq
sudo apt-get install yq
</code>

<code bash>
openstack image show IMAGE1 -c properties -f yaml |yq '.properties.direct_url'
</code>

Filtrer sur les clefs (keys)
<code bash>
cat meta/runtime.yml | yq '.action_groups | keys'
# Ou
cat meta/runtime.yml | yq -r '.action_groups | keys[]'
</code>

Mofifier un fichier YAML 

avec yq :
* https://www.baeldung.com/linux/yq-utility-processing-yaml
* https://dev.to/vikcodes/yq-a-command-line-tool-that-will-help-you-handle-your-yaml-resources-better-8j9

En Python et autres :
* https://stackoverflow.com/questions/63581308/edit-yaml-file-with-bash

Avec Ansible :
https://github.com/kwoodson/ansible-role-yedit

Exemple non-specific tag
<code yaml>
configuration: !include config.d/*.yaml
</code>

Reférences
<code yaml>
vars:
  webapp:
      version: &my_version 1.0
      custom_name:
          - "ToDo_App"
          - *my_version
</code>


=== Exemple Python

<code python>
import yaml
                                                     
with open("filter.yml", "r") as yamlfile:
    #filters = yaml.load(yamlfile, Loader=yaml.FullLoader)
    
    filters = yaml.safe_load(yamlfile)
</code>   
    

== Échapper certains caractères

=== Passer un fichier YAML à une API REST qui fonctionne en JSON

<code bash>
cat inv.yaml |sed -e 's/$/\\n/' -e 's/"/\\"/g' |tr -d '\n'
</code>


==== Exemple YAML vers JSON

''inv.yaml''
<code yaml>
---
server1:
  hosts:
    server1.inf.acme.lan:
  vars:
    SET_LINGERING_LIST:
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "false"
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "true"

type_host:
  hosts:
    server1.inf.acme.lan: {}
  vars:
    type: host
</code>

<code bash>
#! /bin/bash
set -euo pipefail

YAML="$(cat inv.yaml |sed -e 's/$/\\n/' -e 's/"/\\"/g' |tr -d '\n')"

curl -v -k -u user1:'P@ssw0rd' -H 'Content-Type: application/json' -X POST https://awx.acme.fr/api/v2/job_templates/81/launch/ -d '{
"extra_vars": {
"foo1": "bar1",
"foo2": "bar2",
"inventory_content": "'"${YAML}"'"

}
}'
</code>


Ou avec un heredoc

<code bash>
YAML="$(cat <<'EOF' | sed -e 's/$/\\n/' -e 's/"/\\"/g' |tr -d '\n'
---
server1:
  hosts:
    server1.inf.acme.lan:
  vars:
    SET_LINGERING_LIST:
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "false"
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "true"

type_host:
  hosts:
    server1.inf.acme.lan: {}
  vars:
    type: host
EOF
)"
</code>


==== En convertissant le YAML en JSON

En passant un JSON dans un JSON en string

<code bash>
#JS="$(ansible-inventory -i inv.yaml --list)"
JS="$(cat inv.js)"

JS="$(echo $JS |sed 's/"/\\"/g')"
URL=''

curl -v -k -u user1:'P@ssw0rd' -H 'Content-Type: application/json' -X POST https://awx.acme.fr/api/v2/job_templates/81/launch/ -d '{
"extra_vars": {
"foo1": "bar1",
"foo2": "bar2",
"inventory_content": "'"${JS}"'"

}
}'
</code>


=== Exemple de yaml

<code yaml>
CRON_HOST_TARGETS:
  all:
    - CRON_FILENAME: crontab_app1
      CRON_USER: app1
      CRONS:
        - DESCRIPTION: "topmem.sh exécuté toutes les heures"
	        SCHED: "0 * * * *"
	        CMDLINE: "~/scripts/topmem.sh >> ~app1/logs/topmem.out 2>&1"

  sc01:
    - CRON_FILENAME: crontab_app1_sc01
      CRON_USER: app1
      CRONS:
        - DESCRIPTION: "archivage de certains fichiers applicatifs"
	        SCHED: "06 03 * * *"
	        CMDLINE: ". ~/scripts/archive_plopsc.sh -l${LINE} >> ~app1/logs/archive_sc_${LINE}.out 2>&1"

        - DESCRIPTION: Performances
	        SCHED: "05 03 * * *"
	        CMDLINE: ". ~/scripts/appexec.sh appenv.sh ${LINE} > /dev/null; ~/scripts/stat_report_run.sh ${LINE} > /dev/null"
</code>




== Pb 

=== Compile Error - Échapper certains caractères

''plop.yml''
<code yaml>
rec:plop:
  stage: plop
  variables:
    ENV: "RECETTE"

prd:plop:
  stage: plop
  variables:
    ENV: "PRODUCTION"
</code>

<code ->
$ cat plop.yml |yq '.rec:plop'
jq: error: syntax error, unexpected ':', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.rec:plop    
jq: 1 compile error
</code>


==== Solution

<code bash>
cat plop.yml |yq '.["rec:plop"]'
</code>