What a cool map of locations that LEO near me

turbofish

Lifetime RDF Contributor
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
197
Reaction score
391
Location
NW Arkansas
Using data from - - - , I have made a map of all locations 75 miles near me that have been pointed out in a couple of days range [June 19-24] where radar was tagged. After creating an SSIS package, I imported all of the data so that now I have 2.3 million rows of data from just those couple of days, weeded out the junk data, put it in a nice little database and ran some queries on it. From there, I exported the results into a csv file and opened up Google Earth to get a map of all of the locations. Later on I can upload my kmz file to my server and provide a link so that you all can play. This one is for NW Arkansas since that is where I am from.

Keeping in mind that I'm at work and got a ton of projects to do, give me a state/county, I will look up the lat/lon of that county and located all of the speed traps that are in a 75 mile range. One thing that I've been playing around with is scrapping my website and starting over and from that, I can provide a place that you can easily download new locations. Trying to insert all 2.3 million rows tended to kill all of my apps that I have used for maps. If you are data inclined, I can also provide the SQL 2016 database that has compressed nicely into a 55 MB file after getting rid of the stuff I couldn't use. Attached is the CSV file.
Edit: I just noticed that the time is off in the file. Guess I didn't account for greenwich mean time-5/6 for my location
GoogleEarth.jpg


How does a data nerd play after working with huge databases all day long? By playing with more data of course.
 

Attachments

Last edited:

nighthawk

Learning to Fly
Beginner User
Joined
Jul 6, 2016
Messages
242
Reaction score
237
This is really neat, would be cool to see how you collected the data.

Might try loading the data up and play around with it, I have experience in signal processing and the like, but always been interested in big data. Never really found a data set that interested me enough to blow some time on it.

If you don't mind me asking, what kind of tools do you use to work on it? I've used R, Python, Julia and did a lot of work with Matlab.
 

turbofish

Lifetime RDF Contributor
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
197
Reaction score
391
Location
NW Arkansas
I can create a csv of all of the data that includes lat/lon and a time column [normal time and epoc] but I'm a MS-SQL kind of guy. I used SSIS to loop through all of the text files and import them into a junk kind of table [all data into one column] and then created a stored procedure to dig out the data that I needed to put the data that I needed. Then I exported just the stuff that was in a 75 mile radius from me and even loaded it into PowerBI for fun.
If you like 'big data' I have other sets as well including stats on every vehicle sold since 1940 [tiny database really] and another fun project that I have been working on for fun is I have weather data for ever county, every day since 1951 until April of this year. Got some more importing to do but that one has 77 million rows of data.
I have a friend who has a daughter that just graduated from MIT in BI stuff. When I would talk to her, she would do all of the R and Python stuff and I would ask how long would it take her to process data. Most of her data sets are in the 10's of thousands of rows, mine start at 2 million and goes up to millions of rows of total data. I was amazed how much longer it would take her to process data than SQL/OLAP. Still want to learn that stuff. Once I loaded all of the GAIA spacecraft info into a database - 1.3 trillion rows of data but then I had no idea what to do with it.
 

infiniti

Premium RDF Member
Advanced User
Premium Member
Joined
Jun 8, 2014
Messages
2,369
Reaction score
2,636
Location
Ohio
Really cool. Would be amazing to have this as a data source / layer for HighwayRadar @ferius

That way, we could display a bit of a heatmap / hotspot area where speed traps typically are as we go...as long as we are wishing for the moon!
 

doubledge

PSL?
Beginner User
Joined
May 14, 2019
Messages
450
Reaction score
412
This is pretty awesome stuff. Very impressive. You should talk with @ferius and @xydrine regarding each of their software projects. This data set might be interesting to integrate into some kind of predictive model.
 

turbofish

Lifetime RDF Contributor
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
197
Reaction score
391
Location
NW Arkansas
The only problem that I'm having is even when I restrict the data to just one day, the whole map lights up no matter what zip code that I use as the base. But it does show hot spots and places to watch out for
 

IPA

Gimme Shelter
Intermediate User
Lifetime Premium Member
Joined
Feb 3, 2019
Messages
519
Reaction score
621
Location
Atl
Cool work. Perhaps there’s also a way to mark breaks in the median or stretches of road without a median wall/fence for possible I/O awareness.
 

BlueV1

"(Don't) Give Hate a Chance" -Jamiroquai
Advanced User
Lifetime Premium Member
Joined
Feb 8, 2016
Messages
3,377
Reaction score
6,315
Location
Western NC
Here's a JBV1 generated heatmap using the last 30 days of my data. Red is Ka, Orange is K, White is Lidar, Blue is CSA alerts.

[Edited to add] Green are X-band but there are so few of those that you can't see them zoomed out like this.
 

Attachments

Last edited:

BlueV1

"(Don't) Give Hate a Chance" -Jamiroquai
Advanced User
Lifetime Premium Member
Joined
Feb 8, 2016
Messages
3,377
Reaction score
6,315
Location
Western NC
Not to diminish anything in this thread; but to show how it already works in JBV1:
Here's a screenshot showing both live CSAs and heat spots from recent CSAs just over the VA border. :)
 

Attachments

turbofish

Lifetime RDF Contributor
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
197
Reaction score
391
Location
NW Arkansas
More maps. I loaded the data into PowerBI and then color coded based on days.
On this one, I zoomed into one area that is known for being a speed trap paradise and could see on 4 different days at around the same location, there were reports.
WithColor.jpg

This one is zoomed out a bit
WithColor2.jpg

Problems with this is I'm using PowerBI with ArcGIS plugin [both are free - PowerBI is from Microsoft] but when attempting to make a web link, the plugin doesn't work.
All done while at work! My boss thinks that a project takes 2 days but since I've done it so many times, got it done in 2 hours.

If you want to see what that looks like, I uploaded the video to my site [YouTube didn't like it]
PowerBI Map
 
Last edited:

turbofish

Lifetime RDF Contributor
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
197
Reaction score
391
Location
NW Arkansas
I got to admit, JBV1 is a really cool way of visually seeing the history. --- data can be a bit much and difficult to follow. In fact, one of the reasons that I've been wanting to buy a valentine one is just for the app! Maybe one day when they have stock in inventory...
 
Last edited:

Discord Server

Latest threads

Forum statistics

Threads
77,522
Messages
1,181,262
Members
19,831
Latest member
imran161
Top