Converting .dat file into SAS file using a SAS script

[liayin]

I use saspy with Jupyter Lab and OnDemand for Academics. Is there a way to read the .dat file from the PC using a .sas file and save a SAS data file onto the PC? The .dat file is too big to be uploaded.

[tomweber-sas]

Hey, I'm not sure what's 'in' your .dat file. Does whatever SAS code you have in your .sas file process this .dat file as you need? I'll assume so. Since your SAS server is remote from your client machine, SAS would need access to that .dat file to process it. Since it's too big to upload (how big is it? how did you try to upload it?), SAS would need to be able to access it from your client machine (or other remote location) to be able to process it w/ the SAS code you have (submitted to the server via SASPy submit() I expect). So, that would require a filename statement to be able to access that file remotely. SAS FILENAME statement has many access methods for accessing different data, as well as many for access remote files. Here's a link with one, but you can see in the list on the left that there are others which can be used remotely too. I don't know if these all work from ODA, but they may vert well work and allow you to remotely access the .dat file you want SAS to process. https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.2/lestmtsglobal/p103pi2vrzn6qhn1e8alrs01jrb7.htm

[liayin]

Hi and thanks for your reply! The SAS code I have in my .sas file processes the .dat file as I need. The .dat file is 3.1 GB, and when I tried to upload it to OnDemand for Academics, I was told that the limit was 1 GB, and my file was too large. Indeed, I was accessing SAS on the server via the SASPy submit() command. I will read through the document for the FILENAME statement and try it out.

[tomweber-sas]

Oh, that's curious. I was reading this doc and it mentions you have a 5G data limit??? It shows some was of checking your quota too. Out of curiosity, how were you uploading the data? One of the ways it talks about, or using SASPy's upload() method? Is there a particular directory on ODA that you have to use to get the 5G worth of space? Sorry, I havn't really used ODA, certainly not to this degree, so I'm not really familiar enough with these particular constraints.

[liayin]

Thanks for the link. The doc did mention a 5G data limit, it also mentions towards the bottom that there is a 1G data limit for single files (I believe). I used the online interface for uploading.

[tomweber-sas]

Ah, I didn't get that far to see that per file limit. What's in the .dat file? Is it a text file you could split into two then concatenate when processing, or more of a binary file that you can't really do that with? Can you process the .dat file in Python, creating a dataframe from it, then use saspy df2sd() to load it into SAS as a data set?

[liayin]

I would have to look into how to split it. It is a 3.1GB file, so I hesitate to open it in Notepad. I can possibly split it in Python, process the smaller files in SAS, and then combine them in Python. Currently I am setting this project aside. I will explore this route if and when I pick this up again. Thanks!

[jld23]

If the .dat plain text, you could split it up into 3-4 segments and then upload each one and process the multiple files.

…

On Jul 28, 2022, at 10:18 AM, Lia Yin ***@***.***> wrote: Thanks for the link. The doc did mention a 5G data limit, it also mentions towards the bottom that there is a 1G data limit for single files (I believe). I used the online interface for uploading. — Reply to this email directly, view it on GitHub <#473 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYDTS3RR5VYXNYJHJBPRADVWKJEDANCNFSM543D54AA>. You are receiving this because you are subscribed to this thread.

[liayin]

It is probably plain test. I will try to split it in Python if and when I return to this project. Thanks!