GoalDescribe big data using the five Vs and give applications, then plan, test and evaluate a small two-table database and query on paper.
New words
big data · very large and complex collections of data that ordinary methods cannot easily processvolume / velocity · how much data there is, and how fast it is producedvariety / veracity · how many different kinds of data there are, and how far the data can be trustedvalue · how useful the data is once it has been made sense of
Explanation
Big data is data in such huge amounts, and so complex, that a spreadsheet or an ordinary database cannot handle it well. It is described by five Vs. Volume is the amount, velocity is the speed at which new data arrives, and variety is the mixture of numbers, text, pictures and sound. Veracity means that the data must be accurate and trustworthy, so wrong values must be found and removed before use. Value means the data is only worth keeping if it is turned into something useful. Weather forecasting, road traffic control, farming and banking all use big data. When the data is about people, they must give permission, and only the facts that are really needed should be kept.
V
Meaning
Question to ask
Volume
how much data
is it too big for a spreadsheet?
Velocity
how fast it arrives
does it keep arriving every second?
Variety
many kinds
numbers, pictures, text, sound?
Veracity
trustworthy
are there wrong readings?
Value
usefulness
what decision can it help?
Worked examples
Step
Working
Readings per hour
10 000 × 60 = 600 000
Readings per day
600 000 × 24 = 14 400 000
Volume and velocity. A country has 10 000 weather stations and each sends one reading every minute. In one day that is 10 000 × 60 × 24 = 14 400 000 readings. The data is large (volume) and never stops coming (velocity), so a spreadsheet would soon be full.
Variety. A city studies its bus service. The data comes from several sources and in several forms: numbers from the tap cards, pictures from road cameras and typed messages from passengers. A single table with fixed fields cannot hold all of these, which is why big-data tools are needed.
B8fx=AVERAGE(B2:B7)
A
B
1
Hour
Temp (°C)
2
1
12
3
2
13
4
3
−999
5
4
14
6
5
15
7
6
14
8
Average of all six
−155.17
9
Average of five
13.6
Veracity. One weather sensor broke and sent −999 °C. If the readings are averaged as they are, the answer is −155.17 °C, which is nonsense. When the impossible reading is removed first, the average of the other five is 13.6 °C. Big data must be cleaned and checked before it is trusted.
Class activity
Mini project on paper (no computer). A school wants a lost-property database. Plan: draw two tables, Items (an Item_ID, what it is, colour, where it was found) and Owners (an Owner_ID, name, class). Choose the primary key of each and add a foreign key to link them. Make: write six invented records. Test: write one query as a grid with at least two criteria, including one OR or NOT, and work out which records it finds. Swap with a partner, who checks your results. Evaluate: does the database meet its purpose, which is to return an item to its owner? Write one thing that works well and one improvement, such as a missing field or a better key.
Practice
1
Match each situation to one of the five Vs: 1) a power company stores a reading from each of 5 million meters every few seconds; 2) a farm mixes soil numbers, drone pictures and spoken notes; 3) a check removes impossible heights from a list of tree sizes; 4) a council uses traffic records to decide where to build a new crossing.
Situation
Which V?
Millions of meter readings every few seconds
?
Numbers, drone pictures and spoken notes
?
Removing impossible tree heights
?
Using traffic records to choose a crossing site
?
1) Volume and velocity (a very large amount arriving quickly). 2) Variety. 3) Veracity. 4) Value.
2
A small club has 40 members. Does it need big data tools for its membership list? Explain.
No. 40 records is a small amount (low volume) that arrives slowly, and a spreadsheet or a small database can store and search it easily. Big data tools are for data too large, fast or varied for those.
3
A bus company wants to keep a record of every journey made with each passenger’s card. Give two things it must do to treat the passengers’ data responsibly.
It should get permission or tell passengers clearly what is collected. It should keep the data safe and keep only the facts it needs, such as the time and stop, and not extra details such as home addresses.
4
A count of yellow cars on a road uses 100 readings, but 12 are wrong because a camera could not see at night. Which V is at risk, and what should be done before the data is used?
Veracity is at risk. The 12 wrong readings should be found and removed or corrected, or the camera should be fixed, before the counts are used.