QST: mm.Blocks and link streets error

[cdkang]

Hello,

I have tried mm.Blocks as below. But I face the error message. Could you let me know how to solve this issue?

blocks = mm.Blocks(tessellation, edges=extended, buildings=buildings, id_name='bID', unique_id='uID')
blocks_df = blocks.blocks  # get blocks df
buildings['bID'] = blocks.buildings_id.values  # get block ID
tessellation['bID'] = blocks.tessellation_id.values  # get block ID

ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat

Also, I have got the similar error when I did "link streets" as below.

streets["nID"] = range(len(streets))
buildings['nID'] = mm.get_network_id(buildings, streets, 'nID', min_size=300, verbose=False)
tessellation = tessellation.merge(buildings[['uID', 'nID']], on='uID', how='left')

ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat

[martinfleis]

Thanks for the report.

Can you provide full traceback (full error message)? I don't know where does this ValueError come from. If you can prepare a minimal reproducible example, that would also help (you can check this guide - https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

[cdkang]

Merry Christmas and Happy New Year

The below is the full taceback for mm.Blocks error.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_16632/682547552.py in <module>
----> 1 blocks = mm.Blocks(tessellation, edges=extended, buildings=buildings, id_name='bID', unique_id='uID')
      2 blocks_df = blocks.blocks  # get blocks df
      3 buildings['bID'] = blocks.buildings_id.values  # get block ID
      4 tessellation['bID'] = blocks.tessellation_id.values  # get block ID

C:\anaconda\envs\geo\lib\site-packages\momepy\elements.py in __init__(self, tessellation, edges, buildings, id_name, unique_id, **kwargs)
    658             )
    659 
--> 660         cells_copy = tessellation[[unique_id, tessellation.geometry.name]].merge(
    661             centroids_tempID[[unique_id, "component"]], on=unique_id, how="left"
    662         )

C:\anaconda\envs\geo\lib\site-packages\geopandas\geodataframe.py in merge(self, *args, **kwargs)
   1376 
   1377         """
-> 1378         result = DataFrame.merge(self, *args, **kwargs)
   1379         geo_col = self._geometry_column_name
   1380         if isinstance(result, DataFrame) and geo_col in result:

C:\anaconda\envs\geo\lib\site-packages\pandas\core\frame.py in merge(self, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator, validate)
   9189         from pandas.core.reshape.merge import merge
   9190 
-> 9191         return merge(
   9192             self,
   9193             right,

C:\anaconda\envs\geo\lib\site-packages\pandas\core\reshape\merge.py in merge(left, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator, validate)
    103     validate: str | None = None,
    104 ) -> DataFrame:
--> 105     op = _MergeOperation(
    106         left,
    107         right,

C:\anaconda\envs\geo\lib\site-packages\pandas\core\reshape\merge.py in __init__(self, left, right, how, on, left_on, right_on, axis, left_index, right_index, sort, suffixes, copy, indicator, validate)
    700         # validate the merge keys dtypes. We may need to coerce
    701         # to avoid incompatible dtypes
--> 702         self._maybe_coerce_merge_keys()
    703 
    704         # If argument passed to validate,

C:\anaconda\envs\geo\lib\site-packages\pandas\core\reshape\merge.py in _maybe_coerce_merge_keys(self)
   1253                     inferred_right in string_types and inferred_left not in string_types
   1254                 ):
-> 1255                     raise ValueError(msg)
   1256 
   1257             # datetimelikes must match exactly

ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat

[cdkang]

I have change type of uID and tried mm.Blocks. But the error was appeared as below.

ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_5416/682547552.py in <module>
----> 1 blocks = mm.Blocks(tessellation, edges=extended, buildings=buildings, id_name='bID', unique_id='uID')
      2 blocks_df = blocks.blocks  # get blocks df
      3 buildings['bID'] = blocks.buildings_id.values  # get block ID
      4 tessellation['bID'] = blocks.tessellation_id.values  # get block ID

C:\anaconda\envs\geo\lib\site-packages\momepy\elements.py in __init__(self, tessellation, edges, buildings, id_name, unique_id, **kwargs)
    687         )
    688         self.buildings_id = buildings_m[id_name]
--> 689         self.buildings_id.index = self.buildings.index
    690 
    691         cells_m = tessellation[[unique_id]].merge(

C:\anaconda\envs\geo\lib\site-packages\pandas\core\generic.py in __setattr__(self, name, value)
   5498         try:
   5499             object.__getattribute__(self, name)
-> 5500             return object.__setattr__(self, name, value)
   5501         except AttributeError:
   5502             pass

C:\anaconda\envs\geo\lib\site-packages\pandas\_libs\properties.pyx in pandas._libs.properties.AxisProperty.__set__()

C:\anaconda\envs\geo\lib\site-packages\pandas\core\series.py in _set_axis(self, axis, labels, fastpath)
    557         if not fastpath:
    558             # The ensure_index call above ensures we have an Index object
--> 559             self._mgr.set_axis(axis, labels)
    560 
    561     # ndarray compatibility

C:\anaconda\envs\geo\lib\site-packages\pandas\core\internals\managers.py in set_axis(self, axis, new_labels)
    214     def set_axis(self, axis: int, new_labels: Index) -> None:
    215         # Caller is responsible for ensuring we have an Index object.
--> 216         self._validate_set_axis(axis, new_labels)
    217         self.axes[axis] = new_labels
    218 

C:\anaconda\envs\geo\lib\site-packages\pandas\core\internals\base.py in _validate_set_axis(self, axis, new_labels)
     55 
     56         elif new_len != old_len:
---> 57             raise ValueError(
     58                 f"Length mismatch: Expected axis has {old_len} elements, new "
     59                 f"values have {new_len} elements"

ValueError: Length mismatch: Expected axis has 586629 elements, new values have 581254 elements

[martinfleis]

I think that this may be caused by a similar issue as reported in #326. The function now expects a 1:1 match between buildings and tessellation but that is likely something we may want to relax a bit.

[cdkang]

Hello,

I think the Block making error came from the below. Let me know how to clean my building shape file that already preprocess function of momepy. Due to the length mismatch, I can not make Block file with momepy.

Collapsed features : 0
Split features : 229
Overlapping features: 77

[martinfleis]

Would you be able to provide a reproducible example? We may be able to fix the issue in the code but right now I am not able to reproduce it. Check this post on how to do that optimally https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.

[cdkang]

Thank you so much for your help.

I would like to email my shape files to your email.

Please let me know your email that you can get my data.

[martinfleis]

Have you tried crafting the example following the blog post? I can as well debug the issue on your data but considering its size, I'd rather not to.

If you are not able to create the example, can you at least send only a small subset of your data that is also causing this issue?

[cdkang]

Thank you. My small data was fine while data of all Seoul building data caused the trouble. Let me check my big data in ArcGIS and other relevant software. I think that the big one needs some polygon cleaning.

[cdkang]

Hello,

I have compared between building shape file and tessellation file of momepy from the building shape.

I found the below tessellation that one building over the line of tessellation unit which has null uID.

I think that it caused the mismatch of numbers between buildings and tessellation. My small data is fine but I want to analyze whole area of Seoul, South Korea that has 660 km2 area for my research.

I am sure that I can solve many issues during applying momepy if I make cleaned and precise tessellation of my building shape files.

Please let me know how to fix it. The black color is building.

[martinfleis]

That is interesting, can you zoom out on this image to see the context a bit? null uID should never happen so you might have found a bug.

[cdkang]

Thank you.

Let me describe my process.
STEP 1. I used preprocess of momepy to clean my building (n=581263).
STEP 2. I used enclosed tessellation of momepy due to longer time of tessellation making.
STEP 3. I saved the enclosed tessellation file as gpkg(n=596051).
STEP 4. I found NULL uID (n=6051). The pink color map is the tessellation with Null uID.

Further I found a tessellation unit without a building. Also, I found that uID for tessellation saved as object type, not float or integral types.

It looks that tessellation function of momepy makes tessellation units even in no building space like the second image.

I add four images below. The second image shows the overlap between a building and a tessellation.
The third image shows a building in a tessellation with Null uID. The final image also shows some odd tessellation.

[martinfleis]

With enclosed tessellation, this is an expected behaviour.

[cdkang]

Thank you. Let me try normal tessellation work after cleaning my input data.

[cdkang]

[cdkang]

One more question.

Do you know the way to make tessellation in ArcGIS Pro or Other software?

I have found that my ArcGIS Pro made the Thiessen file that has the same number of my building shape file.

In this case, we need extra work to cut the boundary of Thiessen file with the convex hull line.

The Thiessen file is working well when I run momepy.Area. The issue of Thiessen file is point-based, not building polygon-based as momepy does.

Please give me your idea. Thanks a lot.

[martinfleis]

You can lookup the original paper and implement the function yourself in arc. I doubt there is a ready to use equivalent now. But even if it would, I bet you would get a very similar outcome and issues since they are not caused by the algorithm but input data.

[cdkang]

Thank you. I will clean my input data first.

[cdkang]

Hello,

One more question.

I am wondering how to make the number of Collapsed Features of my building input data to ZERO value with ArcGIS Pro or other spatial software.

[martinfleis]

Collapsed features are those that are too small. When the tessellation algorithm calls buffer(-.4) (default for the shrink attribute), they collapse to an empty geometry. You should understand what they represent, because they surely don't represent buildings as to collapse, they have to be smaller than 0.8 meters in width.

[cdkang]

OK. Thank you so much!!!

I found that the preprocess function of momepy makes the number of collapsed features ZERO.

[cdkang]

Hello,

I made the tessellation that has the same number of buildings. It takes six hours.

But, still, I face the error "ValueError: Length mismatch: Expected axis has 586629 elements, new values have 581254 elements" when
I use mm.Block.

Could you let me know how to solve this error? Is there any way to clean street file? I doubt street file caused the trouble.

[martinfleis]

I'll need to get my hands on a sample data that is causing this error to be able to properly debug that. From the code, I am not sure why that happens. If you could get a tiny part that still causes this error and share it that would be great! I am surely unable to debug it on the whole Seoul.

[cdkang]

Thank you.

The small data is fine.

Let me check my data more because I have to make Blocks of the whole of Seoul.

[cdkang]

Hello,

I have tested to make Blocks with momepy.

Making blocks of small areas was working well while all Seoul areas work faced the below error.

Do you have any solution to make better data of buildings, streets, and tessellation? Also, I am wondering how to set tolerance in extened_lines.

ValueError: Length mismatch: Expected axis has 586629 elements, new values have 581254 elements

[martinfleis]

I'll need to find this bug and fix it. It is on my to-do list but I can't promise any timeline to that.

[cdkang]

Hello,

I always appreciate your help for momepy.

I have made block shapefile using building and street/road data for all Seoul area.

Could you let me know how to use the shapefile in momepy with tessellation file made in momepy?

[martinfleis]

I am not entirely sure what your question is. Sorry, can you try to expand and explain your use case?

[cdkang]

Thanks a lot. I will describe my work in detail soon.

[cdkang]

Hello,

I have erased road and street shapefile in Seoul area shapefile.

And then, I captured the polygon where buidlings were contained.

The final outcome would be the blocks of Seoul areas.

Now I want to measure diverse index of blocks with momepy after connecting with tessellation.

[martinfleis]

Hi, we have just merged a fix that should resolve the length mismatch error in Blocks. Can you try it on your case study, if it actually fixes it? You need to install development version of momepy.

pip install git+https://github.com/pysal/momepy.git

[cdkang]

Thanks a lot.

But installing development version of momepy faces the below error.

Running command git clone -q https://github.com/pysal/momepy.git 'C:\Users\cdkla\AppData\Local\Temp\pip-req-build-mojugjb2'
ERROR: Error [WinError 2] 지정된 파일을 찾을 수 없습니다 while executing command git clone -q https://github.com/pysal/momepy.git 'C:\Users\cdkla\AppData\Local\Temp\pip-req-build-mojugjb2'
ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?

[martinfleis]

As the error message says, you need to have git installed. See https://git-scm.com/downloads.

[cdkang]

Thank you. It is a great learning process for me.

I have tested your suggestion. It is working well.