Advertisements

Archive for the ‘Flujos de trabajo /workflows’ Category

Visualizing Tweets!

2016/03/04

How about a quick visualization of tweets in CartoDB?

Hey guys, the way we used to visualize is changing on a  daily basis so it’s time to catch up!!!. Let’s add a timestamp and play!!

tweets-visualization-cartoDB-20160304

We will take a look at this dataset in depth shortly so be aware of our schedule!

Regards,

Alberto CONCEJAL
MSc GIS and remote sensing

Advertisements

Creating value through Open Data

2016/02/19

The benefits of Open Data are diverse and range from improved efficiency of public administrations, economic growth in the private sector to wider social welfare

(Source: http://www.europeandataportal.eu/)

Performance can be enhanced by Open Data and contribute to improving the efficiency of public services. Greater efficiency in processes and delivery of public services can be achieved thanks to cross-sector sharing of data, which can for example provide an overview of unnecessary spending.

The economy can benefit from an easier access to information, content and knowledge in turn contributing to the development of innovative services and the creation of new business models.

Social welfare can be improved as society benefits from information that is more transparent and accessible. Open Data enhances collaboration, participation and social innovation.

clip_image004

The economy can benefit from easier access to information, content and knowledge in turn contributing to the development of innovative services and the creation of new business models.

For 2016, the direct market size of Open Data is expected to be 55.3 bn EUR for the EU 28+. Between 2016 and 2020, the market size increases by 36.9%, to a value of 75.7 bn EUR in 2020, including inflation corrections. For the period 2016-2020, the cumulative direct market size is estimated at 325 bn EUR.

picture_1

New jobs are created through the stimulation of the economy and a higher demand for personnel with the skills to work with data. In 2016, there will be 75,000 Open Data jobs within the EU 28+ private sector. By 2020, this number will increase to just under100,000 Open Data jobs. Creating almost 25,000 new direct Open Data jobs by 2020.

picture_2

Public sector performance can be enhanced by Open Data. Greater efficiency in processes and delivery of public services can be achieved thanks to cross-sector sharing of data, providing faster access to information. The accumulated cost savings for the EU28+ in 2020 are forecasted to equal 1.7 bn EUR.

picture_3

Open Data results in efficiency gains as real-time data is used that enables easy access to information that improves individual decision-making. Three case studies are assess in more detail: how Open Data can save lives, how it can be used to save time and how Open Data helps achieve environmental benefits. For example, Open Data has the potential of saving 7000 lives a year by providing resuscitation earlier. Furthermore, applying Open Data in traffic can save 629 million hours of unnecessary waiting time on the roads in the EU.

picture_4

Economic benefits are primarily derived from the re-use of Open Data. Value is there. The question is how big?

The European Union has adopted legislation to foster the re-use of Open (Government) Data. The expected impact of this legislation combined with the development of data portals, is to drive economic benefits and further transparency. Economic benefits are primarily derived from the re-use of Open Data. Value is there. The question is how big?

Thus, the European Commission, within the context of the launch of the European Data Portal, wished to obtain further evidence of the quantitative impact of re-use of Public Data Resources. A study was carried out with the aim to collect, assess and aggregate all economic evidence to forecast the benefits of the re-use of Open Data for all 28 European Member States and the ETFA countries, further referred to as EU 28+, for the period 2016-2020.

Direct benefits are monetised benefits that are realised in market transactions in the form of revenues and Gross Value Added (GVA), the number of jobs involved in producing a service or product, and cost savings. Indirect economic benefits are i.e. new goods and services, time savings for users of applications using Open Data, knowledge economy growth, increased efficiency in public services and growth of related markets.

The market volume exhibits the totality of the realised sales volume of a specific market; the value added. A distinction can be made between the direct market size and the indirect market size. Together they form the total market size for Open Data. For 2016, the direct market size of Open Data is expected to be 55.3 bn EUR for the EU 28+. Between 2016 and 2020, the market size is expected to increase by 36.9%, to a value of 75.7 bn EUR in 2020, including inflation corrections. For the period 2016-2020, the cumulative direct market size is estimated at 325 bn EUR.

In 2016, there will be 75,000 Open Data jobs within the EU 28+ private sector. By 2020, this number is forecasted to increase to just under 100,000 Open Data jobs. This represents a 32% growth over a 5-year period. Thus, in the period 2016-2020, almost 25,000 new direct Open Data jobs will be created.

Based on the forecasted EU28+ GDP for 2020, whilst taking into account the countries’ respective government expenditure averages, the cost savings per country can be calculated. The accumulated cost savings for the EU28+ in 2020 are forecasted to equal 1.7 bn EUR.

The aim of efficiency is to improve resource allocation so that waste is minimized and the outcome value is maximised, given the same amount of resources. Open Data can help in achieving such efficiency, The study offers a combination of the insights around the efficiency gains of Open Data and real-life examples. Three exemplar indicators are assessed in more detail: how Open Data can save lives, how it can be used to save time and how Open Data helps achieve environmental benefits. For example, Open Data has the potential of saving 1,425 lives a year (i.e. 5,5% of the European road fatalities). Furthermore, applying Open Data in traffic can save 629 million hours of unnecessary waiting time on the road in the EU.

The majority of studies performed previously are ex-ante estimations. These are mostly established on the basis of surveys or indirect research and provide for a wide range of different calculations. No comprehensive and detailed ex-post evaluations of the materialised costs and benefits of Open Data are available. Now that governments have defined Open Data policies, the success of these initiatives should be measured. The study offers several recommendations for doing so.

The report goes into further detail on how Open Data has gained importance in the last several years. Furthermore, the report provides insight into how Open Data can be used, and how this re-use differs around Europe. These insights are used to develop a methodology for measuring the value created by Open Data. The resulting values are presented in a graphical way, providing insight in the potential of Open Data for the EU28+ up to 2020.

 

(Source: http://www.europeandataportal.eu/)

 

Tipologías usuarios Madrid Río. Estadísticas y tendencias interesantes

2015/08/21

Después de 6 semanas en Madrid, no ha habido un solo día en que no tuviera que meterme en Madrid Rio o bien para ir a trabajar, para llevar a la niña a la guardería, volver a casa o simplemente para pasear… Madrid Río se ha convertido en la espina dorsal de mis comunicaciones por la ciudad. Una obra con la que originalmente estaba en profundo desacuerdo (por su planificación y ejecución) se ha convertido en, de alguna manera en el eje que articula mis movimientos.

2015-08-21 08.38.20

Para la multitud de personas a las que como a mí, les ha sorprendido esta infrastructura en frente de sus casas hay posibilidad de encontrarse en el mismo metro cuadrado a alguien corriendo, alguien andando, una pareja con un carrito de niños, unos amigos en patines, alguna persona en bici… unos más rápido y unos más lento, todos hemos de convivir en un trazado de unos 7,5km de largo y no más unos metros de ancho, con lo que unas mínimas normas de circulación se imponen.

Partiendo de la máxima de que el peatón tiene prioridad, no se debería pensar que este pueda hacer de su capa un sayo y moverse a su antojo por el recorrido. Otra máxima debe ser que las bicicletas respeten un límite máximo de velocidad (así como la gente en patines, patinetes, segways, etc).

Mi punto de partida ha sido medir desde un mismo punto la pasada de los usuarios y tipificarlos de acuerdo a su sexo, edad aproximada, tipo de deporte que practican y si estaban ubicados de manera correcta en el recorrido de manera que pudieran interactuar de manera normal con los otros usuarios, minimizando al posibilidad de encontronazos, golpes, caídas, etc. Entendiendo como ‘correcto’ si los usuarios circulan por su derecha.

51243949b

He tipificado a 100 usuarios en dos momentos diferentes del día y en el mismo lugar, para poder estableceer comparaciones. Ahí van los datos y posteriormente los resultados del análisis y algunas preguntas abiertas para cuando haya más tiempo o más interés.

  • #1 Avenida de Manzanares 204/ Madrid RIO 20 de Agosto 2015 entre las 16:28 y las 17:45. 35º centígrados

estadisticas-01
estadisticas-01B

sexo: 0= varón, 1=mujer
tipo: 0=andando, 1=corriendo, 2=bicicleta, 3=otro (segway, patín, patinete, etc.)
resultado: 0=correcto, 1=incorrecto

edad mediana= 28 años
moda sexo=hombre
moda tipo deporte=bicicleta
densidad usuarios=78 usuarios/hora

porcentaje posición correcta: 72%
CORRECTO

correlación sexo-corrección?= 0.28, débil
correlación edad-corrección?= -0.29, hay correlación negativa (débil)
correlación tipo deporte-corrección?= -0.33, hay correlación negativa (débil)
(ver post sobre correlación de variables)

(…)
r=1, correlation is PERFECT
0.75<r<1, correlation is STRONG
0.5<r<0.75, correlation is MODERATE
0.25<r<0.5, correlation is WEAK
<0.25, almost NO correlation, both variables are hardy related
(…)

Resumiendo, a esta hora de la tarde, las 4 y pico del mes de agosto con unos 35 grados celsius, la densidad es de 78 personas a la hora, de las cuales el 72% circula de manera correcta.

El perfil tipo de usuario a esta hora es el de VARON, CICLISTA, 28 AÑOS, POSICIÓN EN LA VÍA CORRECTA

Hay una correlación débil entre sexo y posición correcta, lo que quiere decir que las mujeres y hombres se ubican de manera incorrecta sin seguir ningún patrón definido o lo que es lo mismo, entre los mal colocados, casi el mismo número eran mujeres que hombres.

tampoco hay una relación clara de correlación de acuerdo a la edad o el tipo con respecto a la corrección o no de la ubicación.

  • #2 Avenida de Manzanares 204/ Madrid RIO 21 de Agosto 2015 entre las 09:20 y las 9:42. 28º centígrados

estadisticas-02
estadisticas-02b

sexo: 0= varón, 1=mujer
tipo: 0=andando, 1=corriendo, 2=bicicleta, 3=otro (segway, patín, patinete, etc.)
resultado: 0=correcto, 1=incorrecto

edad mediana= 35 años
moda sexo=hombre
moda tipo deporte=bicicleta
densidad usuarios=273 usuarios/hora

porcentaje posición correcta: 90% 

CORRECTO

correlación sexo-corrección?= 0.08, débil
correlación edad-corrección?= -0.07, hay correlación negativa (débil)
correlación tipo deporte-corrección?= -0.22, hay correlación negativa (débil)
(ver post sobre correlación de variables)

(…)
r=1, correlation is PERFECT
0.75<r<1, correlation is STRONG
0.5<r<0.75, correlation is MODERATE
0.25<r<0.5, correlation is WEAK
<0.25, almost NO correlation, both variables are hardy related
(…)

Resumiendo, a esta hora de la mañana, las 9 y pico del mes de agosto con unos 28 grados celsius, la densidad es de 273 personas a la hora, de las cuales el 90% circula de manera correcta.

El perfil tipo de usuario a esta hora es el de VARON, CICLISTA, 35 AÑOS, POSICIÓN EN LA VÍA CORRECTA

Hay una correlación casi inexistente entre sexo y posición correcta, entre los mal colocados, hay casi el mismo número eran mujeres que hombres, tampoco hay una correlación de acuerdo a la edad o el tipo con respecto a la corrección o no de la ubicación.

Ahora dejo algunas pregutas en el aire, es siempre el perfil tipo el de varón en bici de mediana edad o por el contrario hay picos horarios o ubicaciones donde este perfil es diferente. Podríamos encontrar alguna correlación mayor entre la posición correcta en el recorrido y alguno de los tipos estudiados?. Hay algún otro tipo (por ejemplo nivel de estudios o algún rango específico de edad) en el que veamos una relación clara con la correcta/incorrecta ubicación?.

El estudio específico de estas correlaciones podría permitir informar adecuadamente a los usuarios a através de paneles informativos y de esta manera reducir los potenciales golpes entre las personas que disfrutan de Madrid Río pero también ayudaría a integrar correctamente a los diferentes grupos de usuarios para que todos disfrutemos más adecuadamente de estas instalaciones.

Espero que te haya parecido interesante, si necesitas o quieres más información o aclaración, no dudes en ponerte en contacto conmigo por email.
Un saludo cordial!.

Alberto Concejal
albertoconcejal [at] gmail.com
MSc GIS

Pearson correlation and GIS

2014/11/28


pearson-01
Do these two variables have a correlation?. To answer this important question first of all we have to know that only if it’s a linear relationship and there are no outliers we can take advantage of Mr Pearson’s correlation statiscal tool.

If i love chocolate, does this mean i have tendency of being chuby? or on the other hand there’s no relationship at all. Let’s figure it out.

For this particular occasion, input data XY are two DTM heights, my guess is the following: if correlation is too big, i may deduce they’re not independent products and one might been created from the other, in other words, we might have tried to cheat and we are using a different source that the one we have stated… In GIS sometimes things are not exactly as expected and there’s need to be assertive and making a plan for discovering this minor issues.

 

 

 

Let’s start from the beginning, if source 1 is the same as source 2, the correlation would be perfect, is this correct?. The answer is yes. r (Person correlation) would be = 1. So yes, if this was asking about chocolate and fleshiness this would be 100% right but this hardly or never happens in real life (direct and no other explanation or variable interaction… why is always so0o complicated?).

pearson-formula

pearson-04

With real data, you would not expect to get values of r of exactly -1, 0, or 1. For example, the data for spousal ages (white couples) has an r of 0.97. Don’t ask me where i got this weird source (well, just in case: http://onlinestatbook.com/2/describing_bivariate_data/intro.html)

age_scatterplot

If i fill source 2 with a random number, the correlation would be almost none accordingly (in this case r=0.17)

pearson-06

Now if we see the diagram of the first two sources and we get the Pearson correlation coefficient (r=0.24) which means the correlation is very weak.

pearson-03

But that was only a very small part of the table (only 30 iterations), so if i do the same calculation out of the +13,000 iterations i really need, i get these figures (by the way, theres no need to use such a complicated formula above, you can use this one in EXCEL: =PEARSON(A1:An;B1:Bn))

pearson-07

So the correlation now its moderate, which makes me deduct at least the sources seem different and i’d need more clues to think my customer might have tried to actually cheat me using the same source for both datasets.

Summarizing:

r=1, correlation is PERFECT

0.75<r<1, correlation is STRONG

0.5<r<0.75, correlation is MODERATE

0.25<r<0.5, correlation is WEAK

<0.25, almost NO correlation, both variables are hardy related

I hope you guys have found this post interesting,
looking forward to hear where could you use it and/or your feedback,

Regards,

Alberto Concejal
MSc GIS

Cool data. What is this?

2013/09/16

cool-data

What is this ‘cool data‘ all about?. When i got my degree in Geography (about 15 years ago) it didn’t make sense at all being interested in something like graphic design, photography, video and all this stuff… but not only interested in my leisure time but also for taking advantage of it, getting a life out of it. When, after three years of working in a plane while taking aerial pictures i got fed up with anything related to Geography (and working 24/7) and started working as a website designer it started making a bit of sense… Only five years later, having worked as multimedia photographer, graphic designer and technical salesman everything got aligned and made all the sense in the world… i wanted to come back to Geography but taking advantage of all this years of doing something else…

It was time to resign and comeback to the university to get a MSc (In Remote Sensing and GIS)… leave everything behind and planning my working life again. After finishing my master’s degree (well, actually they were two!) i received a call from South Africa. Are you Alberto?. Hell yes, I am!. We have just received your CV and we are interested in offering you a job dealing with something (i didnt know about) called ‘Geovisualization’. Again, what is this all about?. Well, its dulcifying, easying up all complex information and offering it in a new way, a much easier way to understand it… This is what i call COOL DATA.

Let’s imagine there’s something our customer doesn’t really understand and its kind of difficult to describe using words… There’s always a way for making it easier. Using Graphic design tools or Video software to recreate a workflow or explain something complex… even adding some music, why not???.

cooldata01

Using Photoshop Macros and a few composition rules we can convert a tricky template in something we, our team and our customer will understand completely.  What about Global Mapper scripts for quick conversions between formats?.

cooldata02

What about including some statistics?

cooldata03

and Spatial analysis?.

cooldata04

And 3D?

cooldata05

Maybe some video?

Is this clear enough?. No?.
Let me explain it to you in English, French or Spanish.

Now, seriously, count on me in case of need of converting complex datasets into COOL DATA.
Please don’t hesitate to contact me.

Alberto Concejal
MSc GIS and Remote Sensing
albertoconcejal (at) gmail.com

Projets éoliens en Loire-Atlantique

2013/09/02

C’est vraiment magnifique l’ouverture des données publiques… Je vais décrire mon itinéraire pour mieux comprendre:

  1. Télécharger des données (Projets éoliens en Loire-Atlantique)
  2. Telecharger DTM (SRTM v4)
  3. Faire Carte d’ombrage (ArcGIS),
  4. Orientations (ArcGIS) et
  5. Inclinations (ArcGIS) pour mieux comprendre l’emplacement des moulins
  6. Faire Carte de densité (en mesurent la puissance du parc)

eolicpark_03

Et maintenant la carte de densité en Global Mapper…

eolicpark_02

Name=Chauvé

Feature Type=Unknown Point Feature
Geometry=Point location: 321341.949 6688237.734 (Lat/Lon: 47° 11′ 11.0775″ N, 2° 00′ 13.7962″ W)
Map Name=projets_eoliens.shp
NOM_PARC=Chauvé
ETAT_AVANC=Permis de construire accepté
NBR_EOL=6
PUISSANCE=12

Et après, avec tous les cartes et toutes les données, faire l’interprétation, voici toutes les moulins du vent en Loire Atlantique, on peut apprécier la concentration au Nord/Nord-Est de la région:

eolicpark_01

Et aussi la rose des vents (http://www.nantes-erdre.fr/statistiques-du-vent-a-nantes):

La rose des vents représentative du secteur d’étude est celle fournie par la station de Nantes-Bouguenais.

Les données ont été recueillies sur une période de 29 années (entre le 1er janvier 1971 et le 31 décembre 2000).

La rose des vents ci-après représente la distribution annuelle des vents (tous mois et toutes heures
confondues).

Les vents sont classés selon trois catégories :
– vents dont la vitesse est comprise entre 5 et 16 km/h (bleu),
– vents dont la vitesse est comprise entre 16 et 29 km/h (vert),
– vents dont la vitesse est supérieure à 29 km/h (orange).

Ces catégories sont ensuite reportées en terme de fréquence pour chacune des 18 directions de la rose des vents située au centre (nord, sud, est, ouest, etc.)

Les vents dominants sont les suivants :

– Les vents de secteur ouest/sud-ouest et sud (directions de 180 à 280°) qui représentent 36,5% des vents, toutes vitesses confondues. Les vents les plus forts de la station (vitesse supérieure à 29 km/h) soufflent majoritairement dans ces secteurs.

– Les vents de secteur nord-est (24,7% des vents) avec une majorité de vents faibles ou moyens (directions de 20 à 80°).

frequence-vent
statistiques-vent0

image

Idéalement il faut exporté raster->vector et faire l’analyses spatiale mais ça va être un outre post !

Ici, quelques liens et information général:

L’energie eolique. http://www.loire-atlantique.fr/jcms/cg1_244375/l-eolien

Documentation: http://data.paysdelaloire.fr/donnees/detail/localisation-des-projets-eoliens/?tx_icsoddatastore_pi1[page]=4&visualization=3

Éolien terrestre ou off-shore, ce mode de production électrique devrait connaître une accélération sans précédent en Loire-Atlantique d’ici à 2020.

L’éolien apparaît comme la principale source d’énergie renouvelable électrique permettant d’atteindre dans les toutes prochaines années un niveau important de production.

C’est pourquoi Le Département a fixé en juin 2010 un objectif ambitieux de puissance éolienne installée à l’horizon 2020 :

  • 600 mégawatts (MW) terrestres
  • 500 MW en mer.

Solar + Shadows analysis on Rue Massillon, Nantes (France)

2013/04/04

This is the beautiful building in from of my house in Nantes… Let’s model it first using Sketchup. Also i have used a non standard style only for visualization purposes…

rue_massillon_02

If we have the North behind us this means we are going to have a lot of light… but so far we are just guessing…

rue_massillon_01

Now we know if for sure. Take a look at the results by M. Capeluto’s Solar Analysis.

rue_massillon_03

If we face South we notice there’s a lot of sun in the other side of the building (every line is a summarized path during a month, being the higher the closer to the summer solstice)

Please take a look at the video prepared by Tomasz Janiak, the developer of the tool.

Hope you guys find it interesting. Please let me know your thoughts.

Alberto

ViewTec’s State-of-the-art 3D visualization

2010/09/14

Good news!!!! They’ve probably released this Geovisualization Engine long time ago (It used to be an ‘in-house’ software) but I have just realized. I worked with this state-of-the-art TerrainView 3D software for six months while I was working as a GIS Specialist in Cape Town (South Africa)… What does this software has that no one of other softwares I know have????. Rain, snow, clouds, wind… Athmosferic conditions!!!!  A very reliable package… I have just uploaded a video rendered directly from a pre-recorded flight path… I will try to make my own while I work with this personal license key they have provided me for one month… Here it is!!. Hope you like it mates!.

http://www.viewtec.net/

Products

ViewTec provides state-of-the-art interactive 3D visualization products andreal-time 3D Earth GIS.

TerrainView™

TerrainView™ is the free entry visualization product of ViewTec. TerrainView™ is an easy to use viewing tool and 3d editor capable of displaying the most popular 3d formats. It offers walk-through capabilities in real-time.TerrainView Extensions

Upgrade the free TerrainView product online to the different professional extensions:

TerrainView-Globe™

TerrainView-Globe™ extends TerrainView with unlimited GIS, multilayered elevation and image data import based on Earth and offers ultra-fast and high resolution rendering for GIS and simulation users.

TerrainView-RemoteControl™

The extension TerrainView-Remote Control™ offers an API to remotely manipulate points of interest and 3D objects of a 3D scene in real-time.

TerrainView-CMAX™

The extension TerrainView-CMAX™ consists in a high-end system for seamless multi-channel, multi-projections and visualizations on multiple screens and walls.

TerrainView-Video™.

TerrainView-Video™ is the extension to produce videos in a resolution higher than 640×480 e.g. PAL, NTSC and HDTV.

Visualizing LAS LIDAR data with sketch-up

2010/07/02

While trying to figure out the way to get a 3D model from raw Lidar data, I first opened my LAS file in Global Mapper, exported it to DXF, imported then into Sketch-up and after recording a few scenes, I saved the animation… this is it!.

Alberto

Using Excel to calculate the RMSE for LiDAR vertical ground control points

2010/06/30

(source: http://dominoc925.blogspot.com/)

The height accuracy of the collected LiDAR data can be verified by comparing with independently surveyed ground control points on hard, flat, open surfaces. It is essentially just calculating the height differences for all the control points and then determining the height root mean squared error (RMSE) or differences. Most LiDAR processing software have the reporting function built-in. However, plain Microsoft Excel can also do the job (except for extracting the elevation from the LiDAR data).

Assuming that you are able to calculate the height differences for all the control points and place in a spreadsheet as shown in the figure below. I have a column of delta Z values in column A.

Then to calculate the RMS value for the elevation differences, I can do the following.

  1. In a cell, type in the formula:= SQRT(SUMSQ(A2:A18)/COUNTA(A2:A18))where A2:A18 are the values from cell A2 to A18 in the spreadsheet. Simply replace these with the actual locations on your spreadsheet.
  2. Press RETURN.
    The RMSE value is calculated.

(source: http://dominoc925.blogspot.com/)