Issues to be covered

  1. JSON
  2. JSON-RPC
  3. REST
  4. cURL
  5. JQ
  6. Client-server architecture
  7. Sieci P2P

Activities

  1. Introduction to the topic of classes, providing teaching materials.
  2. Overview of client-server architecture and REST architecture
  3. Overview of URLs
  4. A brief discussion of the cURL and JQ libraries
  5. Sending a request via cURL to the poznan.pl API
  6. Requesting a request by cURL to the openweathermap.org API using the JQ library.

Materials for classes

Introduction tasks

Task 1 - cURL / JSON

  1. Send the cURL request to the http://jsonplaceholder.typicode.com/posts/1 page. What does the result look like? How is it different from the data in (known from the introduction to computer science) xml format?

    $ curl 'http://jsonplaceholder.typicode.com/posts/1'

Task 2 - cURL / JSON /XML

  1. Take a look at the API of poznan.pl

    http://www.poznan.pl/api/
  2. Using the cURL tool, get information about events in XML format from the API’s web page poznan.pl.

    $ curl 'http://www.poznan.pl/mim/public/ws-information/?co=getCurrentDayEvents' 
  3. Use the cURL tool to get a list of streets in the JSON format from the API’s page poznan.pl

    $ curl 'http://www.poznan.pl/featureserver/featureserver.cgi/ulice/all.json'
  4. Limit the previous call to the first 100 streets

    $ curl 'http://www.poznan.pl/featureserver/featureserver.cgi/ulice/all.json?maxFeatures=100'

    If you want to see the returned JSON with 100 streets in Poznan in a more readable form, you can use the jq command

    $ curl 'http://www.poznan.pl/featureserver/featureserver.cgi/ulice/all.json?maxFeatures=100' | jq
  5. The program jq is a powerful tool, you can use it to search and filter data in JSON format. For example, the list of street names can be built in the following way

    $ curl 'http://www.poznan.pl/featureserver/featureserver.cgi/ulice/all.json' | jq '.features[].properties.a3'

Task 3 - cURL & JQ

  1. Browse the API on the page

    https://openweathermap.org/api

    What data and in what format can you get on this site?

  2. Register on the site

    https://openweathermap.org/api

    Download the private key.

  3. Use the cURL tool to get from https://openweathermap.org/ the weather in the city with the first letter such as the first letter of your surname.

Assessment tasks

Mandatory

Basic tasks are checked automatically, hence it is important to send to the appropriate address and maintain the indicated format.

The results of tasks are available in the text file at https://kino.vm.wmi.amu.edu.pl/results/dtin/######.txt, where ###### is a six-digit index number. Global list of results is available at https://kino.vm.wmi.amu.edu.pl/results/dtin.txt

Task 4.1 - Factoring the index number (1p.)

Send to http://kino.vm.wmi.amu.edu.pl:6080/dtin/z4.1/######, where ###### is a six-digit student index number, the PUT request containing a JSON file in the message body for which the following jq filter will return the sum of the prime factors of your index number (for example, for the number \(60 = 2 \cdot 2 \cdot 3 \cdot 5\), the result should be 12):

[.factors[].value] | add

Do not forget to set the correct Content-Type header.

Tasks 4.2 and 4.3 - jq filters (4p.)

Send to http://kino.vm.wmi.amu.edu.pl:6080/dtin/z4.2/###### or http://kino.vm.wmi.amu.edu.pl:6080/dtin/z4.3/######, where ###### is a six-digit student index number, the POST request containingjq filter, which for specified files will return the appropriate response.

Description of the file structure

The input JSON file shows the history of certain events (block) and the history of the value of a certain parameter (hashrate). Its general structure is as follows:

{
   "blockHistory": [1519739287, 1519739455, 1519739710 ...],
   "hashrateHistory": [
      {
         "hr": 105355000000,
         "time": 1519739200
      },
      {
         "hr":104900000000,
         "time":1519739600
      }
      ...
   ]
}

Both the values in the blockHistory array and the time attribute are given as Unix time, i.e. as the number of seconds since the beginning of the 1970.

Test file

The task will be checked on the following file.

The expected result of the filter operation:

Task 4.2 - Manipulating objects (2p.)

For the input file, we want to get a common history of hashrate and block values sorted chronologically. The output file is to have the following structure:

[
   {
      "time": 1519739200,
      "type": "hr",
   },
   {
      "time": 1519739287,
      "type": "block"
   },
   {
      "time": 1519739455,
      "type": "block"
   },
   {
      "time": 1519739600,
      "type": "hr"
   },
   {
      "time": 1519739710,
      "type": "block",
   },
   ...
]

Task 4.3 - Advanced jq - variables and functions (2p.)

For the input file, we want to obtain information about the * hashrate * parameter value recorded immediately before each event, sorted chronologically. The output file is to have the following structure:

[
   {
      "lastHr": 105355000000,
      "time": 1519739287
   },
   {
      "lastHr": 105355000000,
      "time": 1519739455
   },
   {
      "lastHr": 104900000000,
      "time": 1519739710
   },
   ...
]
Tips
  • Use the variable mechanism to capture the main stream for use at different filter locations.
  • Create a function that calculates the hashrate value at a given time and use it when creating a result stream.
  • Use the -f parameter to save the filter in a separate file. This will make it easier for you to prepare a solution.

Extra

Task D4 - Web crawler (5p.)

Write a simple Internet Crawler, which for a given topic page on Wikipedia, will display a list of all topics from the See also section. The search should be performed recursively.

For example, for the topic https://en.wikipedia.org/wiki/Online_chat, in addition to the topics directly included in See also such as: https://en.wikipedia.org/wiki/Chat_room or https://en.wikipedia.org/wiki/Instant_messaging the results should also include these See also topics (and so on) for example: https://en.wikipedia.org/wiki/Social_media or https://en.wikipedia.org/wiki/Media_psychology.